Skip to content

feat(rees): add performance guardrails and regression telemetry #1814

Description

@JSONbored

Context

REES needs measurable performance targets so future analyzer work does not reintroduce timeout races, unbounded fanout, or high resource usage. The #1805 timeout fix made history fail open, but the service still needs broader performance regression protection.

This issue adds explicit guardrails, tests, and safe telemetry for REES runtime performance.

Parent: #1499

Deliverables

  • Define target budgets for REES profiles, including request reserve time before engine transport timeout.
  • Add lightweight performance regression tests with fake slow analyzers and fake slow external services.
  • Add safe per-request/per-analyzer telemetry: elapsedMs, analyzer status, partial/degraded status, cap reason, external call counts, cache hit rate, and endpoint category counts.
  • Add Sentry tags/context for request id or trace id, repo, PR number, head SHA prefix, requested analyzers, analyzer, phase, subcall, timeout budget, elapsed time, and partial status.
  • Add guardrails for request body size and schema validation before analyzer execution.
  • Document how to read REES degradation/partial signals during production incidents.

Requirements

  • Telemetry must exclude secrets, tokens, request bodies, raw diffs, raw prompts, comments, and private config values.
  • Repo name, PR number, head SHA prefix, release, elapsedMs, counts, categories, and reason codes are allowed.
  • Performance tests should be deterministic and not depend on real network services.
  • Guardrails should degrade or reject safely with useful status, not crash the service.

Acceptance criteria

  • A slow analyzer cannot cause the engine to abort before REES returns partial/degraded output.
  • Sentry issue context is immediately attributable to analyzer, phase/subcall, endpoint category, timeout budget, and partial status.
  • Oversized or malformed requests are handled before expensive analyzer work begins.
  • Regression tests cover the timeout/degraded path and safe telemetry exclusions.

Validation

  • Focused REES performance/timeout tests.
  • Sentry context/scrubbing tests.
  • Engine enrichment-wire test confirming degraded/partial REES responses are consumed without caller-side abort.

Ownership

Maintainer-only. No gittensor:* labels.

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:featureGittensor-scored feature linked to a feature issue — scores a 0.25x multiplier.

    Projects

    Status
    Done

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions