Repository navigation
roadmap(observability): vendor-neutral execution tracing for agents and DevLoop #55
Description
Context
mctl already emits or plans the telemetry needed to understand agent execution, but the pieces are tracked as unrelated issues:
mctlhq/mctl-gitops#902— OpenTelemetry Collector as the mctl telemetry gateway;mctlhq/mctl-gitops#903— backend/approach comparison: Traceway, Tempo, Langfuse, Phoenix, SigNoz, Grafana Agent Observability/agento11y;mctlhq/mctl-agents#195— end-to-end execution traces across Temporal / Argo / model / MCP / GitHub;mctlhq/mctl-agent#38— OpenTelemetry tracing and token-usage metrics inmctl-agent;mctlhq/mctl-agent#97— correlate execution traces with Loki, incidents and the mctl MCP surface.
Until now these were only "related work" under mctlhq/.github#18 (agent platform) and mctlhq/.github#48 (agent FinOps). Nothing owned the sequence, so the Collector was never deployed and the backend decision stayed open. This epic owns that sequence.
Goal
Make every mctl execution — DevLoop stage, incident diagnosis, MCP call — observable as one correlated trace, on a vendor-neutral producer boundary:
mctl workloads → OTLP → OpenTelemetry Collector → one or more replaceable backends
Invariants:
- OTLP / OpenTelemetry is the only application-side contract. No backend-specific SDK enters
mctl-agentormctl-agents. - Trace identity reuses the execution identity from
mctl-agents#196(execution_id, Temporal workflow/run id, WorkItem, agent/profile version). No second correlation scheme. - Private prompts, completions and Telegram message bodies stay out of telemetry by default.
- The trace backend is a view. Durable evidence (
mctl-agents#199) and the cost ledger (.github#48) do not depend on trace retention.
Phases
Phase 0 — telemetry gateway
mctl-gitops#902. Deploy the Collector, define the OTLP endpoints workloads use, and pin the resource attributes every producer must set. Fan-out to backends is a Collector concern only.
Phase 1 — backend bake-off
This phase is split at the offline/live boundary.
- Phase 1a — comparison harness:
mctl-gitops#903— complete. Frozen rubric, paper screen, representative DevLoop fixtures, generalized exporter/fan-out harness and ADR skeleton are in place. - Phase 1b — live bake-off and decision:
mctl-gitops#1280— open. Deploy the viable candidates, fan out representative traces through the Collector, measure a declared soak window, fill the rubric from observations and close the ADR.
The candidate set is now Traceway, Tempo, Langfuse, Phoenix, SigNoz, plus Grafana Agent Observability / agento11y as an instrumentation/UX approach. Grafana must be evaluated in two distinct shapes:
- self-managed
agento11y/OTel telemetry → mctl Collector → Tempo/Grafana, preserving the vendor-neutral telemetry boundary; - optional Grafana Cloud Agent Observability for conversation/session, cost/eval and guard UX, explicitly scoring Cloud dependency, privacy/data residency, pricing and portability.
The live bake-off must also prove whether agento11y covers the real claude-agent-sdk DevLoop runtime path, not only interactive Claude Code/Codex-style sessions.
Phase 2 — end-to-end execution traces
mctl-agents#195 and mctl-agent#38. Every DevLoop stage and every mctl-agent diagnosis emits spans carrying the canonical execution identity, including model calls with token usage. This phase is the prerequisite for .github#48 Phase 1 (one DevLoop with cost attribution).
Phase 3 — cross-system correlation
mctl-agent#97. Link traces to Loki log streams, incidents and MCP operations so an operator can move from an alert to the execution that caused it and back.
Current state (2026-10-13)
Verified against GitHub and the live cluster on 2026-10-13. The canonical graph is roadmap/epics/observability.yaml; this section is a snapshot of it.
| Work item | Required | Issue | Live evidence |
|---|---|---|---|
Telemetry gateway, mctl-gitops#902 |
yes | closed 2026-09-11 | Collector in monitoring. Every producer exports to otel-collector.monitoring:4318. |
Attribute catalog, mctl-docs#116 |
yes | closed 2026-09-22 | docs/reference/telemetry-attributes.md |
Offline backend evaluation and rubric, mctl-gitops#903 |
yes | closed 2026-09-22 | ADR 0001, frozen rubric, fixtures |
DevLoop execution tracing, mctl-agents#195 |
yes | closed 2026-10-11 | trace 164ac97d8d7264326c406240d263dcc1 |
mctl-agent tracing, mctl-agent#38 |
yes | closed 2026-09-23 | live since mctl-agent 2.5.0 (gitops#1828), trace 3b54b8b36a8a3334173dc4cad2458e83 |
MCP span context, mctl-api#285 |
yes | closed 2026-10-10 | trace_id in operation results |
Cross-system correlation, mctl-agent#97 |
yes | closed 2026-10-13 | incident argo-mctl-agents-implement-dae24112-1791756910 → workflow → trace 431486061c652efb376a860d9d656f25 → ERROR span → Loki. mctl_get_incident returns the trace_id (mctl-api 4.82.0). |
Live Tempo validation and ADR decision, mctl-gitops#1280 |
yes | open | soak in progress, see below |
- Phase 0, telemetry gateway: done.
- Phase 1a, offline selection: done.
- Phase 1b, live validation:
mctl-gitops#1280is the only open required item.- Scope (owner decision 2026-10-10, ADR "Bake-off scope amendment"): the live round is Tempo-only, monolithic on R2. The paper screen already ruled out deploying the other candidates, including the Grafana/
agento11yshapes described under Phase 1 above. Under the frozen decision rule, Tempo is the default verdict unless live evidence gives a concrete reason against it. - Declared soak: from 2026-10-10T11:55Z to 2026-10-23T23:59Z, teardown 2026-10-24.
- Real producers admitted by recorded amendments: the
mctl-agentsDevLoop (gitops#1788) andmctl-agent(gitops#1860). - Mid-window record (mctl-gitops#1280, 2026-10-13): 1,840 of 1,840 spans accounted for, no stop rule triggered, 0 leak-check hits.
- Adopting Tempo permanently is a separate owner-gated change after the verdict, not a promotion of the eval deployment.
- Scope (owner decision 2026-10-10, ADR "Bake-off scope amendment"): the live round is Tempo-only, monolithic on R2. The paper screen already ruled out deploying the other candidates, including the Grafana/
- Phase 2, producer instrumentation and live proofs: done (
#195,#38). Prerequisitesmctl-gitops#1332(token redaction) andmctl-agents#196(execution identity) are closed. - MCP propagation: done (
mctl-api#285). - Phase 3, cross-system correlation: done (
mctl-agent#97).
Not required by the roadmap, but open: mctl-agent#139 step 3, promoting the catalog attributes that are now shipped. The catalog still marks gen_ai.* as reserved and says no MCTL service emits it.
Remaining critical path
#1280 soak (to 2026-10-23 23:59Z)
│
▼
rubric cells + decision rule → ADR Decision
│
├──► owner-gated adoption change (if Tempo is adopted)
▼
teardown of observability-eval (2026-10-24) → verify nothing left running
│
▼
close #1280 → close this epic → roadmap lifecycle
Previous snapshot (2026-09-24), kept for history
Current state (2026-09-24)
The roadmap is in implementation, and the old 2026-09-11 "not started" snapshot is obsolete.
- Phase 0 — telemetry gateway:
mctl-gitops#902closed 2026-09-11. The OTel Collector is the vendor-neutral tracing gateway. - Phase 1a — bake-off harness:
mctl-gitops#903closed 2026-09-22. Offline comparison assets, fixtures, fan-out/evaluation harness and ADR skeleton landed. The live/operator portion was deliberately split out. - Phase 1b — live bake-off:
mctl-gitops#1280open. This is the current critical-path work item: deploy viable candidates, run the declared soak, compare real traces, select the architecture in the ADR and remove non-selected candidates. On 2026-09-24 Grafana Agent Observability/agento11ywas added to this live comparison as candidate/approach F. - Phase 2a —
mctl-agentinstrumentation code:mctl-agent#38closed 2026-09-23 via PR MCP servers: CIMD primary, DCR fallback restricted to the portal callback, full scope for DCR clients #137. Pipeline andgen_ai.*spans plus token metrics are implemented using standardOTEL_*configuration. - Phase 2b —
mctl-agentlive rollout:mctl-agent#139open. Release the code, enableotel.enabledin-cluster, verify real spans, then promote telemetry catalog attributes. - Phase 2c — DevLoop end-to-end tracing:
mctl-agents#195open, but the code/offline portion merged in PR #466. Trace propagation Temporal → Argo → pod, model/tool/GitHub/artifact spans, policy events, export-time redaction and failure isolation exist. It remains open for live exported evidence and one real investigation + implementation trace. - Collector correctness blocker:
mctl-gitops#1332open. Current redaction matches the substringtokenand masks GenAI token-count attributes; fix before using token/cost visibility as bake-off evidence. - Phase 3 — cross-system correlation:
mctl-agent#97open; depends on live end-to-end traces. It will connect incident/GitHub issue → workflow → trace/failed span → Loki and expose trace identity through operational/MCP surfaces.
Current critical path
#1332 fix Collector token redaction
│
├───────────────┐
▼ ▼
#1280 live bake-off #139 agent rollout
│ │
└──────┬────────┘
▼
#195 live DevLoop proof
│
▼
#97 trace ↔ Loki ↔ incident
The immediate roadmap focus is therefore #1280, not creating another backend-comparison epic. Grafana/agento11y is evaluated inside that existing decision path.
Repository ownership
| Repository | Ownership |
|---|---|
mctl-gitops |
Collector deployment, backend deployment, retention, dashboards |
mctl-agents |
DevLoop span instrumentation and identity propagation across Temporal / Argo |
mctl-agent |
diagnosis / remediation instrumentation, Loki and incident correlation |
mctl-api |
MCP-side span context and, later, trace links in operation results |
mctl-docs |
telemetry contract, attribute catalog, operator guide |
Relationship to other roadmap items
.github#48agent-finops consumes Phase 2 spans; the cost ledger stays mctl-owned regardless of the backend chosen here..github#18agent-platform supplies the execution identity and profile versions that spans carry..github#20context-platform later attaches ContextSnapshot references to the same traces.
Acceptance criteria
- An OpenTelemetry Collector receives OTLP from at least
mctl-agentsandmctl-agentin the preview cluster. (Since 2026-10-10 / 2026-10-11, see #1280.) - A written ADR selects the backend architecture, with offline comparison evidence from
mctl-gitops#903and live bake-off evidence frommctl-gitops#1280, including the Grafana/agento11ypath. (Open: the verdict comes after the soak. The Grafana/agento11ypath was screened on paper only, per the 2026-10-10 Tempo-only scope decision.) - One complete DevLoop is visible as a single correlated trace (canonical criterion,
roadmap/epics/observability.yamlsuccessCriteria; the roadmap is authoritative). Met bymctl-agents#195, trace164ac97d8d7264326c406240d263dcc1in the #1280 Tempo eval backend, a single trace for DevLoopdev-loop-mctlhq-mctl-api-454:-
Traced execution: the
mctl-agents-workerTemporal activities (RunActivity:submit_and_wait×3) with oneargo.workflowspan per stage (investigate,implement,approve, allSucceeded). Under them, themctl-agents-investigatorandmctl-agents-implementerpods:invoke_agent,chat <model>spans with token usage,execute_tool, and git/GitHub spans (clone, push, issue comment, PR create). -
Reviewer and shepherd activity are not traced, by design. Reviewers run as the
claude-review.ymlGitHub Action, outside the OTLP producer boundary. The approve and shepherd stages appear in the trace as orchestration spans (the worker'sargo.workflow mctl-agents-approvespan with its Argo workflow name and phase). Their content is in the existing orchestration and audit evidence: Temporal workflow history, Argo workflow status, mctl-api audit records, and the PR review and merge history on GitHub. No reviewer or shepherd instrumentation will be added to satisfy the wording below. -
Superseded wording (historical, 2026-09)
"One complete DevLoop run is visible as a single trace: investigator → implementer → reviewers → shepherd, with model calls and token usage per span."
This prose predates the deployed tracing design and was broader than the canonical roadmap criterion. It is superseded on 2026-10-13 by the canonical criterion above and is no longer an open acceptance criterion.
-
- One
mctl-agentincident diagnosis is visible as a trace linked to the triggering alert and its Loki context. (mctl-agent#97, accepted by the owner 2026-10-12.) - Span attributes follow a documented mctl attribute catalog mapped onto OTel GenAI semantic conventions. (Checked against 1,840 real spans on 2026-10-13: the extra keys are upstream
gen_ai.*/error.type, as the catalog requires. The catalog's own status column is stale, seemctl-agent#139.) - No private prompt/completion or message payload is stored in traces by default. (Leak check over all 1,840 stored spans, 0 hits, method on #1280. The poisoned-fixture check is still pending under #1280.)
- Replacing the backend requires a Collector configuration change only. (Producers depend only on OpenTelemetry/OTLP and export to the Collector. Tempo is one named Collector exporter.)
Non-goals
- Making the trace backend the ledger for cost or the store for execution evidence.
- Adding Traceway, Langfuse or any other vendor SDK to application code.
- Replacing Loki / VictoriaMetrics / Grafana for logs and platform metrics.
- Full-fidelity prompt recording for debugging; that is a separate, opt-in decision.
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsTodo
Activity
mashkovd commentedon Sep 10, 2026
Polish 2026-09-11
Read-only pass over the body (no comments yet), the 5 native sub-issues, every issue referenced in text (mctl-agents#196/#197/#198/#199/#60/#149, .github#18/#48/#20), project #1 fields,
mctl-gitops/ROADMAP.md,mctl-agents/docs/adr/007, the merged PR history of mctl-gitops, mctl-agents, mctl-agent, mctl-api and mctl-docs, and an org-wide code search for OpenTelemetry / Collector / backend references.Stale
Status=Todoin project docs: update org profile status to public/self-service #1. Nothing to untick.Orphan
Gap
mctl-docsis listed under Repository ownership with no child at all. Suggested child in mctl-docs:docs(observability): mctl telemetry contract and span attribute catalog (execution identity + OTel GenAI semconv mapping).mctl-apiownership row has no child. "MCP-side span context and, later, trace links in operation results" is Phase 3 work that mctl-agent#97 explicitly leaves to the MCP surface. Suggested child in mctl-api:feat(mcp): propagate OTel span context through MCP calls and return trace_id in operation results.docs/adr/directory; only mctl-agents keeps ADRs (005–008). Where the backend ADR lives needs deciding before #903 can close.Conflict
internal/diagnosis; that package was removed as dead in mctl-agent#123 (2026-09-05). The live model path isinternal/skill/builtin/llm_diagnosis.go, and token metrics would sit next to the existinginternal/metricspackage. The child body needs a refresh before it is executable.execution_idtoday. ADR-007 explicitly keeps traces and fix(claude-review): tell the reviewer it cannot run checks #196 out of scope, but it does already fixdefinition_version,profile_versionandrelease_revisionon every run, so those three attributes can be pinned in the Phase 0 catalog without waiting on fix(claude-review): tell the reviewer it cannot run checks #196. The remaining identity fields need one declared owner.go.opentelemetry.io/otelonly as an indirect dependency), no Collector manifest exists in mctl-gitops, and mctl-agents#275 exports Temporal SDK metrics to Prometheus only. Starting point is zero traces; recorded in the new Current state section.Blocker
Next executable step
mctl-gitops#902 — deploy the Collector. It has no upstream dependency, every other child needs an OTLP endpoint to exist (the #903 bake-off fans out behind it, #195/#38 need somewhere to send spans), and it is the only place the ROADMAP.md contradiction can be resolved in code (amend the roadmap in the same PR). The #902 PR should also pin the first resource attributes (
service.name, environment,definition_version/profile_version/release_revisionfrom ADR-007) so the docs child has a concrete starting point.Proposed, needs a decision
mctl-gitops/ROADMAP.mdin the #902 PR, or hold Phase 0. Recommendation: declare it met; DevLoop and the incident responder are multi-hop today.mctl-gitops/docs/adr/(deployment-owning repo) or put it in mctl-docs. Recommendation: mctl-gitops, next to the manifests it governs.internal/diagnosis, point atinternal/skill/builtin/llm_diagnosis.go+internal/metrics), mctl-gitops#903 (title + candidate E SigNoz), mctl-agent#97 (#903 is a comparison, not a Traceway POC).execution_idschema and that spike(mcp-gateway): validate identity, per-tool policy and tenant isolation before migration #195 consumes it, so the circular "depends on" is broken in one direction.mashkovd commentedon Sep 11, 2026
Decision 2026-09-11 on item 5 (identity ownership): mctlhq/mctl-agents#196 owns the
execution_id/ execution-context schema; #195 and this epic consume it. Dependency edge #195 ← #196 added, so the cycle is broken in that direction. Phase 0 (#902) can proceed withservice.name/ environment attributes now and adopt the identity attributes when #196 lands.mashkovd commentedon Sep 11, 2026
Decisions applied 2026-09-11
Executed the decisions taken on the
Polish 2026-09-11items 1–4 (item 5 was already decided in the comment above).mctl-gitops/ROADMAP.md(lines ~35 and ~149) in the same change and pin the first resource attributes (service.name, environment,definition_version/profile_version/release_revisionfrom ADR-007).mctl-gitops/docs/adr/. Comment on spike(observability): compare Traceway, Tempo, Langfuse, Phoenix, SigNoz and Grafana Agent Observability mctl-gitops#903 (spike(observability): compare Traceway, Tempo, Langfuse, Phoenix, SigNoz and Grafana Agent Observability mctl-gitops#903 (comment)); the directory is created in the PR that lands the ADR. #903 retitled to name all five candidates, and SigNoz moved from the 2026-09-08 comment into the body as candidate E (fan-out diagram, deliverable list and ADR section updated).trace_idin operation results. Depends on #902, mctl-agents#195/fix(claude-review): tell the reviewer it cannot run checks #196, mctl-agent#97 and mctl-docs#107.internal/diagnosis; points atinternal/skill/builtin/llm_diagnosis.goandinternal/metrics; exports over OTLP to the #902 Collector;Depends on: mctlhq/mctl-gitops#902.Depends on: mctlhq/mctl-agents#195.No code, branches or PRs were created;
ROADMAP.mdis amended by the #902 PR, not here. Next executable step remains mctlhq/mctl-gitops#902.mashkovd commentedon Sep 11, 2026
Portfolio hygiene — 2026-09-12
Recommended portfolio priority: P1 — active observability foundation.
This snapshot supersedes the stale
Current state (2026-09-11)section above.Live state correction
mctlhq/mctl-gitops#902(OpenTelemetry Collector telemetry gateway) closed 2026-09-11.mctlhq/mctl-gitops#903remains open and is the backend-decision track.mctlhq/mctl-agents#195andmctlhq/mctl-agent#38must produce a representative real execution trace.Next executable order
#195+ producer side as needed).#903backend comparison rather than benchmarking empty/synthetic infrastructure only.mctlhq/mctl-agent#97cross-system incident/Loki/MCP correlation.Portfolio rule
Do not make the backend bake-off the critical path to emitting traces. OTLP/Collector is already the contract; backend choice must remain replaceable and must not block producer instrumentation.
mashkovd commentedon Sep 12, 2026
Producer-side input from
mctlhq/mctl-telegram#617(Phase 3 of the enterprise MCP roadmap.github#35), shipped in mctl-telegram 0.66.0.tg.mctl.ainow records how every MCP call arrived, inaudit_logsand mirrored to its structured log line:edge_request_id(the Cloudflare ray),edge_route(portal/direct),mcp_method,mcp_name,protocol_version. Captured once per request ininternal/edgectxand read fromctxat the audit write, so there is a single capture point for whatever consumes it next.The OTel half of #617 is deliberately not done there and belongs to this epic. mctl-telegram has no tracer —
opentelemetryis an indirect dependency only — and adding one locally would have created the second correlation scheme this epic's invariants forbid. When Phase 0's Collector lands and a tracer reaches this service, the span attributes are already settled and read the same values the audit row carries:mctl.edge.request_idedge_request_idmctl.edge.routeedge_routemcp.method/mcp.namemcp_method/mcp_namemcp.protocol_versionprotocol_versionThree measured facts worth having in this epic's context before anyone designs trace correlation across the Cloudflare boundary:
Cf-Rayarrives on both routes — the zone is proxied, so a direct consumer call gets one too. It identifies a request, not a route; the discriminator isCf-Worker.server/discover, thentools/call), each with its own ray. A per-call span must expect the pair.mcp.mctl.aileg cannot be tied to the subrequest leg today. Nothing in the arriving request carries the client-leg ray, and Cloudflare's per-request dataset is unavailable on this zone's Free plan (parentRayNameabsent from the schema,rayNameentitlement-denied, adaptive groups empty, no Logpush). So cross-layer correlation with Cloudflare evidence is by principal and time; a paid plan would change that, which makes it a cost decision rather than an engineering one.Also worth carrying over: these fields are headers as received — evidence for reading a trail, never an authorization input. A direct caller cannot forge
edge_routeon this zone (Cloudflare strips client-suppliedCf-*, measured), but that is a property of the current Cloudflare configuration rather than of mctl.Full measurements: mctl-telegram#617 — header table, worked example, forgery test, ray result.
mashkovd commentedon Nov 1, 2026
Final state — completed
Closing this epic after the canonical roadmap reached completion:
roadmap/epics/observability.yamllifecycle changed fromactivetocompletedin chore(roadmap): complete observability epic #296 (merge69d0f1e).The informational
HierarchyUnexpectedChilddiagnostic for mctlhq/mctl-docs#107 is non-blocking housekeeping and is not part of this epic's completion gate.