feat(observability): tracing bake-off sandbox, candidate manifests, fan-out, teardown and runbook (config slice of #1280) #1355

Context

Split from #1280 by owner decision (2026-09-24). The first implementer run on #1280 ended no-commits: the proposal mixed reproducible gitops/config work with live-cluster measurement, and the implementer correctly refused the live half.

This issue owns the reproducible code/config slice: everything that can be written, rendered and tested offline and merged inert. #1280 keeps the live bake-off: opening the sandbox, soaking, measuring, scoring, deciding and tearing down, all run by a human operator.

Scope

All of it is off by default. With otelCollector.eval.enabled: false, the rendered output of platform-gitops/bootstrap must be byte-identical to today's.

1. Sandbox namespace and manifests. ResourceQuota + LimitRange for observability-eval inside the existing otelCollector.eval.enabled guard, driven by new eval.quota / eval.limits values:

   • the caps are persistentvolumeclaims: 4 and requests.storage: 40Gi;
   • a render test asserts them.

2. Candidate deployments. One manifest set per candidate that survives the Stage A paper screen (from the spike(observability): compare Traceway, Tempo, Langfuse, Phoenix, SigNoz and Grafana Agent Observability #903 rubric), each behind its own per-candidate flag under otelCollector.eval, with a pinned chart/image version:

   • no candidate is enabled in this issue;
   • candidate F (agento11y) is prepared only as the self-managed shape;
   • the Grafana Cloud shape needs no in-cluster manifests; its evaluation belongs to spike(observability): run the live tracing backend bake-off and decide #1280.

3. Collector fan-out config. The OTel Collector exporter/pipeline fan-out to the enabled candidates, rendered only for candidates whose flag is on, plus a render test:

   • with all flags off, there is no fan-out;
   • with one flag on, there is exactly one extra exporter and the production pipeline is unchanged.

4. Redaction regression test. A test that renders the collector and checks two things:

   • gen_ai.usage.{input,output,cache_read_input,reasoning}_tokens survive redaction (otel-collector: redaction drops all GenAI token-count attributes; deployment.environment uses a deprecated key #1332);
   • github_token, authorization, api_key and vault.token are still blocked.

   Wire it into validate-manifests.yml. If otel-collector: redaction drops all GenAI token-count attributes; deployment.environment uses a deprecated key #1332 is still unfixed, the test is committed as an expected failure that names otel-collector: redaction drops all GenAI token-count attributes; deployment.environment uses a deprecated key #1332.

5. Teardown automation. Two parts:

   • otelCollector.eval.teardownAfter is enforced by a CI check that fails once the date has passed while eval.enabled is still true;
   • a documented one-commit teardown (flags off) that removes every candidate and the namespace.

6. Runbook. docs/runbooks/tracing-bake-off.md is the operator procedure spike(observability): run the live tracing backend bake-off and decide #1280 executes:

   • preconditions check;
   • open the sandbox and set the teardown date (≤ 14 days);
   • declare the soak window and trace volume before measuring;
   • where each rubric cell's measurement comes from;
   • the scoring and ADR promotion steps;
   • teardown and verification that nothing is left running.

Out of scope (stays in #1280)

• Enabling the sandbox or any candidate in the live cluster.
• The soak, every measurement and every rubric score.
• The ADR decision.
• The actual teardown.

Acceptance

• With every eval flag off, helm template platform-gitops/bootstrap is unchanged versus main. This is checked in CI.
• Render tests cover the quota/limits, per-candidate enablement, fan-out on/off, and the redaction rules.
• The teardown-date CI check exists and is tested for both states.
• The runbook covers every step spike(observability): run the live tracing backend bake-off and decide #1280 needs and references each flag by name.
• No non-null rubric score cell is committed.

Related

• spike(observability): run the live tracing backend bake-off and decide #1280: the live bake-off and decision (human/operator acceptance).
• spike(observability): compare Traceway, Tempo, Langfuse, Phoenix, SigNoz and Grafana Agent Observability #903: the rubric, fixtures and fan-out harness.
• otel-collector: redaction drops all GenAI token-count attributes; deployment.environment uses a deprecated key #1332: collector token redaction.
• Roadmap epic roadmap(observability): vendor-neutral execution tracing for agents and DevLoop .github#55.