Repository navigation
spike(observability): run the live tracing backend bake-off and decide #1280
Description
Context
mctlhq/mctl-gitops#903 was investigated as a single programme: rubric, paper screen, trace fixtures, collector fan-out, an evaluation namespace, deploying several candidates, a live soak, scoring and an ADR. One implementer PR cannot carry that, and several steps need a merge and a live cluster between them.
The split follows the point where evidence stops being reproducible offline. #903 keeps the rubric, the paper screen, the representative trace fixtures and the fan-out harness, all of which are code and render tests. This issue owns everything that needs candidates actually running.
Scope
- Create the evaluation sandbox namespace.
- Deploy each candidate that survives the paper screen.
- Turn on the collector fan-out to the deployed candidates.
- Measure over a fixed, declared soak window: trace reconstruction, search and correlation, token/tool/error/cost visibility, privacy, producer isolation.
- Fill every cell of the frozen rubric from a measurement, then promote the decision into the ADR and the runbook.
- Tear down every non-selected candidate and the sandbox.
Acceptance
- Every rubric cell is filled from an observation, and any cell that could not be measured says so instead of being scored.
- The soak window and the trace volume are declared before the measurements, not chosen afterwards.
- One backend is selected in the ADR with the evidence that decided it.
- Nothing from the bake-off is left running.
Depends on
mctlhq/mctl-gitops#903— the rubric, the fixtures and the fan-out harness this measures with.
Related
- Roadmap epic
mctlhq/.github#55
Candidate/approach F — Grafana Agent Observability / agento11y (added 2026-09-24)
Carry candidate F from #903 into the live comparison rather than treating the offline screen as sufficient.
Evaluate two shapes separately:
-
Self-managed Grafana-native path
- supported agent/runtime instrumentation via
agento11ywhere feasible; - OTLP traces/metrics → existing mctl OTel Collector;
- Tempo/Grafana as the trace UI/store path;
- metadata-only capture by default.
- supported agent/runtime instrumentation via
-
Grafana Cloud Agent Observability path
- conversation/session and coding-agent UX;
- token/cost/eval visibility;
- optional guards only as defense-in-depth;
- explicitly measure Cloud dependency, data egress/residency, pricing and portability.
The acceptance bar is not "it works with interactive Claude Code". The bake-off must state whether agento11y can observe the production claude-agent-sdk DevLoop execution path without becoming a Grafana-specific application dependency. If it cannot, record that limitation and score the approach accordingly.
mctl runtime policy/approval (mctlhq/mctl-agents#197/#198) remains authoritative; Grafana guards are not a replacement governance boundary.
Preconditions / current blockers
Before token/cost evidence is considered valid:
- fix
mctlhq/mctl-gitops#1332: Collector redaction currently masksgen_ai.usage.*_tokensbecause the key contains the substringtoken; - use the merged producer instrumentation from
mctlhq/mctl-agent#38/ PR feat(labs): Hook B — labs-trading-data MCP service + openclaw wiring #137 andmctlhq/mctl-agents#195/ PR fix(labs/pfeifenpatenschaft-backend): wire CNPG dbSecret #466 when live rollout is available; - coordinate with
mctlhq/mctl-agent#139for the mctl-agent cluster rollout.
The bake-off may prepare namespaces/manifests while those prerequisites are being closed, but the final scoring must use correctly exported live telemetry.
Activity
mctl-agents commentedon Sep 24, 2026
mctl-agents issue-investigator has analyzed this issue and created a proposal:
https://github.com/mctlhq/mctl-gitops/tree/main/platform-gitops/agents-state/mctl-gitops/proposals/issue-1280-spike-observability-run-the-live-tracing/
Status:
proposed— pending human approval. Reviewrequirements.md,design.mdandtasks.md, then approve: signal this issue's DevLoopWorkflow (mctl-apiPOST /api/v1/agents/dev-loop/dev-loop-mctlhq-mctl-gitops-1280/approve, orpython -m orchestrator.temporal.cli approve dev-loop-mctlhq-mctl-gitops-1280 --approver <your-identity>) — the workflow flips.status.yamltoacceptedvia themctl-agents-approveoperation and runs the Tier 2 implementer. If no DevLoopWorkflow is running for this issue (pre-Temporal proposal), run themctl-agents-approveoperation directly withservice=mctl-gitops slug=issue-1280-spike-observability-run-the-live-tracing.The approver is recorded in
.status.yamland in the gitops commit, and the implementer refuses a proposal whosecontrol.requires_human_approvalcarries no named approver — so prefer the mctl-api endpoint, which takes the approver from the authenticated caller rather than from what the caller types.chore(agents): investigate #1280 2026-09-24
mashkovd commentedon Sep 24, 2026
Approved by the owner on 2026-09-24 via the DevLoop approve signal, after the proposal was amended with binding cost and egress constraints in #1351 (
c9f3143):See the "Owner constraints" section of
design.md.mashkovd commentedon Sep 24, 2026
Re-scoped by owner decision (2026-09-24). The first implementer run on this issue ended
no-commits, because the proposal mixed reproducible config work with live-cluster measurement.issue-1280-spike-observability-run-the-live-tracingstays atneeds-triageas the record of that run.Precondition #1332 (token redaction) still gates valid token/cost evidence.
mashkovd commentedon Oct 10, 2026
Synthetic span proof — PASSED (soak window starts 2026-10-10T11:55Z)
Tempo eval deployed by #1764 (Tempo 2.10.8, single binary, R2 bucket
tempo-traces). Runbook step 4.3, one submission ofotel-trace-fixture(workflowotel-trace-fixture-proof-l8jgm, Succeeded 11:48:57Z):tempo_distributor_spans_received_totaltempo_ingester_traces_created_totaltempo_discarded_spans_totaltempo_ingester_blocks_flushed_total/failed_flushes_totaltempo-eval)devloop.runtraces frommctl-agentsa9486b83e86d32dd2b8942eb731a1cf6Declared soak (ADR 0001 amendment, accepted with #1764): window from this proof until 2026-10-23 23:59 UTC; volume synthetic only, < 5,000 spans total; real producer traffic none. Teardown date 2026-10-24.
Defect found during the proof: the collector labels its Tempo exporter telemetry
otlp_grpc/eval-tempo, so the two collector-side alerts could never fire. Fix: #1765.Merge pull request #1788 from mctlhq/docs/1280-amend-soak-devloop-pro…
mashkovd commentedon Oct 11, 2026
Soak record — observed volume reconciled (2026-10-11 06:10Z)
The soak-start comment above declares "real producer traffic none". That was true at 11:55Z. The ADR's Producer amendment: DevLoop traces (recorded before any producer emitted) then enabled the
mctl-agentsDevLoop producer at 2026-10-10 20:18:43Z. Counters have read 252 accepted / 252 sent since 20:41Z, so this note accounts for every span instead of leaving 240 vs 252 unexplained.Full enumeration of Tempo eval since 2026-10-10 11:00Z: 23 traces, 252 spans, matching
tempo_distributor_spans_received_total= 252 exactly.service.nameotel-trace-fixture-proof-l8jgmdevloop.run, 12 spans each)mctl-agentsadmins-mctl-agents-worker-exec-base-service…-worker-exec-base-service-5bfcc57847-mqgqxThe 12 producer spans are 6
RunActivity:submit_and_waitspans and 6argo.workflow mctl-agents-shepherdsubmit spans. They come from three DevLoops whose Temporal runs started before enablement but scheduled shepherd activities after it:trace_idb2cfbbca71c55a264912fd7f9de3002cdev-loop-mctlhq-mctl-api-53745a0ca72449b49d90d899259188691b5dev-loop-mctlhq-mctl-api-5369be69f5847d13f44928c6cbeb9fe4afcdev-loop-mctlhq-mctl-agent-162Notes:
service.name. Spans exported after fix(base-service): stop overriding producers' service.name #1791 should carrymctl-agents-worker.Declared vs observed
Stop rule (amendment), checked 06:05Z — no trigger
tempo_discarded_spans_total= 0.The soak window and teardown date are unchanged: 2026-10-23 23:59Z, teardown 2026-10-24. Nothing here requires restarting the soak.
Merge pull request #1860 from mctlhq/docs/1280-mctl-agent-producer-am…
mashkovd commentedon Oct 13, 2026
Soak record: mid-window checkpoint (2026-10-13 12:30Z)
The soak is still in progress. The window, volume, stop rules and teardown date are unchanged: the window ends 2026-10-23 23:59Z and teardown is 2026-10-24. This is not a verdict. It records evidence in a durable place before teardown removes Tempo.
Volume: every span accounted for
Source: Tempo
/api/searchover 2026-10-10 11:00Z → now, plus/api/v2/traces/<id>for every hit.Result: 43 traces, 1,840 spans. That equals
tempo_distributor_spans_received_total, Collectorotelcol_receiver_accepted_spans{receiver="otlp"}, andotelcol_exporter_sent_spans{exporter="otlp_grpc/eval-tempo"}, all 1,840.The Collector pods have been up for about 72.6h, so the counters cover the whole window without a reset.
otel-trace-fixture-proof-l8jgm(service.name=mctl-agents)admins-mctl-agents-worker-exec-base-service, pre-#1791)mctl-agents-worker178,mctl-agents-investigator708,mctl-agents-implementer691)mctl-agentincident tickets (service.name=mctl-agent)Producer cap: 1,600 of 500,000.
Accounting notes:
mctl-agentamendment (docs(adr): record mctl-agent producer amendment for #1280 soak #1860, merged 2026-10-12 20:12Z) lists 2 traces / 7 spans as observed before the record. A third ticket trace,5dfa4b390bc96ff4af79d89819a43c0(4 spans, 20:01Z), also predates the merge. It is inside the same cap and the same rules; this note only completes the enumeration.mctl-agents-implementruns at 21:04Z and 21:31Z (9cf3d81c,883394eb) were not submitted by a traced DevLoop, soMCTL_TRACE_REQUIRE_PARENT=truekept them inert, as declared.Stop rules (DevLoop + mctl-agent amendments): none triggered
tempo_discarded_spans_totalincreaselive_traces_exceeded,trace_too_large,unknown_error)TempoEvalCollectorExportFailures,TempoEvalCollectorDroppedSpans,OtelCollector*)send_failed= 0,enqueue_failed= 0,refused= 0,queue_sizemax 0 of 5,000Operations (rubric
operationssources)tempo-0: 0 restarts, uptime about 3d.tempodb_compaction_errors_totalandtempodb_retention_errors_total: 0.tempo_receiver_refused_spans: 0.observability-eval-quota: used 1 pod, 100m/512Mi requests, 1/1Gi limits, 0 PVCs, against a hard limit of 4 pods, 1/1536Mi requests, 2/2Gi limits.TempoEval*andOtelCollector*rules are present in vmalert.Trace reconstruction and search (rubric
trace_reconstruction)mctl-agenttraces each have one root and 0 orphans.{span.mctl.workflow.id="dev-loop-mctlhq-mctl-telegram-757"}{span.mctl.repository.name="mctlhq/mctl-api"}{span.mctl.issue.number=586}{span.gen_ai.agent.name="issue-investigator"}{status=error}{span.mctl.argo.workflow.name=~"mctl-agents-implement-.*"}{resource.service.name="mctl-agent" && span.mctl.ticket.type="resource_limit"}{span.gen_ai.request.model="claude-opus-5-5"} | sum(span.gen_ai.usage.output_tokens) > 0431486061c652efb376a860d9d656f25(80 spans) and 3.5s for551bb5d8b7569d152be3ddc5801f8c9(252 spans).164ac97d8d7264326c406240d263dcc1(mctl-agents#195 evidence).431486061c652efb376a860d9d656f25(mctl-agent#97 evidence). The chain is incidentargo-mctl-agents-implement-dae24112-1791756910→ workflowmctl-agents-implement-dae24112→ ERROR spanImplementerOrphanedSubagent→ Loki[trace]anchor.mctl_get_incidentreturns thistrace_idwithtrace_source=workflow_auditsince mctl-api 4.82.0.3b54b8b36a8a3334173dc4cad2458e83(cpu_throttle).Privacy: leak check method and result
status.messageand every span name of all 1,840 spans was scanned with these patterns:Bearer …or any key containingauthorization;gh[pousr]_…/github_pat_…;sk-…/sk-ant-…;eyJ….….);password|secret|api_key = …;AKIA…;prompt,completion,input/output.messages,tool.call.arguments/result,message.text/body/content,*.body,command,args,arguments.gen_ai.usage.input_tokens,output_tokens,cache_read.input_tokens,cache_creation.input_tokens) are present as integers, not masked. Thetokensubstring in those keys is a usage count, not a credential.ImplementError/ "proposal scope too large for one PR".Backend neutrality
mctl-agentandmctl-apigo.mod, andmctl-agentspyproject.toml/uv.lock, contain onlygo.opentelemetry.io/*/opentelemetry-*and OTLP exporters. There is no Tempo, Grafana or vendor SDK.http://otel-collector.monitoring.svc.cluster.local:4318:admins-mctl-agent, themctl-agents-worker-exec/-implementDeployments, and themctl-agents-investigate/-implementWorkflowTemplates.otlp_grpc/eval-tempo, next todebug.Still to measure before the verdict
security_privacy: runbook §6 also asks for "a look at what Tempo stores for the poisoned fixturedevloop-trace-redaction.json". It has not been submitted yet: the only fixture submission used the defaultdevloop-trace.json. One submission is about 240 spans and fits the declared synthetic volume (at most once a day, under 5,000 in total). Submitting it needs an operator.data_ownership_portability): not yet checked directly.The rubric cells, the decision rule and the ADR Decision section are filled only after 2026-10-23 23:59Z.
mashkovd commentedon Oct 13, 2026
security_privacy synthetic measurement: poisoned fixture (run once)
Run: Argo workflow
otel-trace-fixture-redaction-v96lz(labelmctl.ai/purpose=issue-1280-security-privacy), fixturedevloop-trace-redaction.json, 2026-10-13 13:26:45Z → 13:27:15Z,Succeeded. Parameters:execution_count=10, 4 spans per trace. The production producers and the collector config were not touched.Volume: 10 traces × 4 spans = 40 spans submitted. Synthetic total is now 280 spans, inside the declared < 5000 budget. This is the only run, so the once-per-day limit holds.
otlp_grpc/eval-temposentdebugsenttempo_discarded_spans_totalDropped or rejected spans: none. Tempo TraceQL
{span.mctl.execution_id="exec-poisoned-redaction-fixture"}returns exactly 10 traces (rootdevloop.run, servicemctl-agents), and fetching all of them gives 40 spans.Redaction result: PASS.
gen_ai.chatgen_ai.prompt.0.content(fabricated prompt text)gen_ai.prompt****,redaction.masked.count=1mcp.tool.callmcp.tool.arguments(fabricated args)mcp.tool.arguments****mcp.tool.callmctl.artifact.note(fakeghp_-shaped string under a neutral key)ghp_****,redaction.masked.count=2on the spangithub.requesthttp.request.header.authorization(fabricated bearer)http.request.header.*/authorization****,redaction.masked.count=1I ran a substring scan over the raw JSON of all 10 stored traces. Each of
FABRICATED,ghp_,Bearer,fabricated-not-a-realand the fake token body appears 0 times. The value-pattern case passing matters most: it shows that a secret-shaped value under an innocuous key is still caught. Thedebugexporter output in both collector pod logs since 13:26Z mentions the execution id 10 times (7 + 3), so the spans did pass through it. The same strings appear 0 times there, and the logs contain no error, drop or refusal lines.Secrets: every planted value was fabricated for this fixture. The
ghp_value is a sequential placeholder (A1b2C3…) that has never been a GitHub token, and the prose values say "FABRICATED, NOT A REAL …" in plain text. No real credential, prompt or customer data was used.Stop rules after the run: cap 1880 / 500,000. No discard increase, no exporter alert, no leak-check hit. The soak continues unchanged to 2026-10-23T23:59Z. I won't run the fixture again unless the owner approves another run.
mashkovd commentedon Oct 22, 2026
Soak closed early: final measurements (cutoff 2026-10-22 17:00Z)
Caveat first. The owner closed the window on 2026-10-22 for operator capacity, 30 h 59 min before the declared end of 2026-10-23 23:59Z. The actual window was 2026-10-10 11:55Z to 2026-10-22 17:00Z, 293 h 05 min or 90.4% of the declared 324 h 04 min. No stop rule had fired and no measurement prompted the decision. It is recorded as a retro amendment in ADR 0001 ("Window amendment"), and the Decision opens with it.
Ingestion and the stop rules (counters at 17:00Z)
accepted_spans7,170 =sent_spanstootlp_grpc/eval-tempo7,170 = Tempodistributor_spans_received_total7,170. Refused 0. Discarded 0, for every reason.send_failed/enqueue_failedhave no series. The exporter queue stayed at 0.release-tail.export-checkspan (2026-10-16 13:47Z, origin not recorded) make 281 of the 5,000 allowed. Producer spans were about 6,890 of the 500,000 cap.TempoEvalObjectStorageErrorswas pending 08:37–08:41 and never fired. No alert fired inside the window.Post-soak checks (read-only)
R2 Parquet readability. The bucket holds 874 objects (30.35 MB) and 218 blocks, 200 of them live. Two blocks were downloaded and decoded with
pyarrow, without Tempo: the newest and the largest (6 traces, 353 spans, compaction level 3). Both arevParquet4. The bucket lifecycle read returned AccessDenied for the token in use, so the 30-day backstop was not re-read.Grafana
tempo-evalsearch over the window, as two 168 h halves:{status=error}Every query shape tried enumerates the same 139 traces, and no query adds a trace.
Loki → Tempo. 50 log lines with
trace_idin 12 h. 4 opened through the derived field (2–45 spans, 1.6–11.2 s).Tempo → Loki.
tracesToLogsV2(namespace, pod, ±1 m) resolves a DevLoop trace's pods to their log lines.Privacy, final scan. Every attribute key and value of every fetched span, the poisoned fixture's traces included, was scanned. The fixture was not rerun. No credential, token or e-mail was found. The only pattern hits are the fixture's own
authorizationkey with value****.gen_ai.prompt.0.content×10 andmcp.tool.arguments×11 are all****. The longest value is 63 characters.Findings that cut against the result
Verdict
ADOPT TEMPO / mctl-native only, 315/500 on the Tempo column (4, 2, 4, 3, 4, 0). This is decision-rule branch 3: Tempo's AI/agent score of 2 does not beat the
openclaw-llm-usagedashboard baseline, also 2, by more than one point. The other candidates were not deployed (scope amendment), so their cells are cited nulls. The ADR, rubric and runbook are updated in the verdict PR.Merge pull request #2071 from mctlhq/docs/1280-soak-verdict
mashkovd commentedon Oct 22, 2026
Closed with the verdict ADOPT TEMPO / mctl-native only (315/500), merged in #2071 (c880817). The soak window was shortened by owner decision for operator capacity: it ended 2026-10-22 17:00Z instead of 2026-10-23 23:59Z, 30 h 59 min short (90.4% of the declared window). This is recorded in ADR 0001's Window amendment and leads the Decision. The evidence is in #1280 (comment).
Follow-ups:
validate-eval-teardown-date.pyturns main red.Merge pull request #2073 from mctlhq/chore/1280-teardown-eval-sandbox
feat(observability): deploy Tempo as the permanent trace backend