Repository navigation
chore(observability): tear down the #1280 Tempo eval sandbox - #2073
Conversation
The #1280 soak closed on 2026-10-22 with the verdict ADOPT TEMPO / mctl-native only. The sandbox was declared disposable, and validate-eval-teardown-date.py turns main red after 2026-10-24, so it goes now in the one commit runbook section 9 prescribes, instead of being promoted in place. Because Tempo was selected, the TempoEval alerts and the R2 bucket stay (step 9.4 and the bucket note). The permanent deployment is a separate, owner-approved change. The real producers' export is switched off as both ADR producer amendments require, so nothing keeps exporting into a Collector that has no trace backend. The render test now asserts the closed state (no eval exporter, datasource, namespace, Application or Loki derived field). The Tempo wiring checks still run against an explicit eval-on overlay. Refs #1280
|
Claude finished @mashkovd's task in 4m 5s —— View job Review complete ✅
Traced the full gating chain for
P3 (nice to have, does not block merge): the module docstring (lines ~39-44) and the No P1/P2 findings. Verdict submitted: APPROVE. |
Antigravity (agy) review — informationalModel: gemini-3.8-flash-high · commit No significant issues found. |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
No P1/P2 findings (1 P3: stale module docstring/T1 comment describing the old sandbox-open committed default, outside the diff hunk, no functional impact). Gating chain traced end-to-end (namespace, Application, collector exporter, Tempo + Loki datasources, teardown-date validator, worker otel env) and all correctly closed/off. Good to merge.
|
Skipping — this commit (52539a1) already has a Claude review verdict. Push a new commit, or check the run log if you expected a fresh pass and the previous run actually failed. |
Antigravity (agy) review — informationalModel: gemini-3.8-flash-high · commit No significant issues found. |
|
Skipping — this commit (d4f1c87) already has a Claude review verdict. Push a new commit, or check the run log if you expected a fresh pass and the previous run actually failed. |
Antigravity (agy) review — informationalModel: gemini-3.8-flash-high · commit No significant issues found. |
The #1280 verdict (ADR 0001) adopted Tempo, mctl-native only, as a separate deployment rather than the sandbox promoted in place. The owner approved the plan on 2026-10-22 (Option A, teardown first; #2073 merged). Tempo 2.10.8 (chart 2.4.0) runs in its own `tracing` namespace, reuses the tempo-traces R2 bucket, keeps 336h and is read-bounded (api_timeout, querier and frontend concurrency) after the soak's unranged read pinned the pod. root-app renders only the Namespace and the Application; all namespaced objects come from the Application's manifestsPath, which avoids the project/namespace sync deadlock of gitops#1925. The collector exports to it over otlp/backend; Grafana gets datasource "Tempo" (uid tempo) and deletes the read-only "Tempo (eval)" leftover via provisioning, since the API refuses. Only the DevLoop workers and admins-mctl-agent export. The emptyDir WAL is an owner-accepted standing risk. The unexplained partial trace-by-id read (5,092 of 6,930 spans) is a documented limitation: no completeness claims or counts from Tempo until explained (docs/runbooks/tempo.md).
Tears down the #1280
observability-evalsandbox in one commit, per runbook §9 with Tempo selected.otelCollector.eval.enabled: false, thetempocandidateenabled: false,teardownAfter: "".validate-eval-teardown-date.pywould otherwise turnmainred on 2026-10-24.otel.enabled: falseon the threemctl-agents-worker*services and onadmins-mctl-agent. Nothing exports into a Collector with no trace backend, as both ADR producer amendments require.vm-rules/tempo-eval-alerts.yamlstays. It is silent without the sandbox, and the adoption PR renames it.tests/test_otel_collector_backends_render.pychecks the closed default: nootlp/eval-exporter, traces go to[debug]only, notempo-evaldatasource, noobservability-evalNamespace or Application, and no Loki derived field. The Tempo wiring checks now render with an eval-on overlay. Mutation check: re-enabling eval in the committed default makes 6 checks fail.The R2 bucket
tempo-traces, its token and Vaultsecret/platform/r2-tempoare kept, because the permanent Tempo reuses them. After the Argo CD prune comes the manual step: delete the "Tempo (eval)" Grafana datasource, then verify with runbook §10.Ordering. This PR is option A of the adoption plan (#2072 draft): tracing is off until adoption lands. Merge it after the verdict PR #2071 and by 2026-10-24, unless the owner picks option B (adoption first).
Refs #1280
🤖 Generated with Claude Code