Skip to content

feat(observability): correlate execution traces with Loki, incidents and mctl MCP #97

Closed
Closed
feat(observability): correlate execution traces with Loki, incidents and mctl MCP#97

Description

@mashkovd
mashkovd
opened on Aug 26, 2026

Context

mctl-agent is the best home for the cross-signal correlation entry point because it already owns incident handling, operational MCP surfaces and structured logging, while mctl-agents owns Temporal/Argo execution state.

Relevant implementation points visible today include:

  • internal/mcp/server.go
  • internal/monitor/*
  • internal/pipeline/*
  • existing structured logging via slog

Related issues:

  • mctlhq/mctl-agent#38 — OpenTelemetry tracing + token usage
  • mctlhq/mctl-agents#195 — end-to-end execution traces
  • mctlhq/mctl-agents#196 — execution identity/runtime context
  • mctlhq/mctl-agents#199 — auditable execution evidence
  • mctlhq/mctl-gitops#902 — OpenTelemetry Collector telemetry gateway
  • mctlhq/mctl-gitops#903 — backend comparison (Traceway, Tempo, Langfuse, Phoenix, SigNoz) behind the Collector; not a Traceway POC

This issue should coordinate with mctl-agents where workflow-side log fields must be added, but the operator/incident/MCP correlation contract should live here.

Producer-side propagation in mctl-agents is owned by mctlhq/mctl-agents#195 and mctlhq/mctl-agents#196 and must not be reimplemented under this issue.

Dependencies

Depends on: mctlhq/mctl-agents#195 (end-to-end execution traces; this issue correlates spans that #195 and #38 emit). Part of roadmap epic mctlhq/.github#55, Phase 3.

Goal

Make it possible to move deterministically between:

incident / GitHub issue
      ↓
workflow execution
      ↓
trace / failed span
      ↓
related Loki logs

The long-term target is that an mctl incident investigator can start from an incident and programmatically discover the failed span and relevant logs without a human manually searching multiple systems.

Scope

Structured log correlation

Add/standardize correlation fields where available:

  • trace_id
  • span_id
  • workflow_name and/or workflow_id
  • run_id
  • incident/ticket ID where applicable
  • GitHub issue/PR metadata where applicable

Use stable field names across mctl-agent and mctl-agents so Loki queries do not depend on service-specific naming.

Workflow → trace → Loki

Define and demonstrate the correlation path:

  1. start from workflow name/ID/run ID
  2. resolve trace ID / relevant execution span
  3. query related logs using trace/workflow identity
  4. identify failed/error spans and associated logs

Document the canonical query/navigation approach independently of the selected trace backend where possible.

Incident/evidence metadata

Where incidents or evidence records already carry workflow/execution metadata, add trace identity without duplicating full telemetry:

execution:
  workflow_id: ...
  run_id: ...
  trace_id: ...

Persist span_id only where it refers to a meaningful root/failure span; do not treat transient child span IDs as durable incident identity by default.

mctl operational / MCP interfaces

Where practical, expose trace_id (and workflow/run identifiers if missing) from operational MCP responses such as workflow/run/incident inspection so callers can pivot directly into observability data.

Do not make MCP responses depend on the trace backend being healthy. Stored/correlated identifiers should still be returned if Traceway/Tempo/etc. is unavailable.

Future investigator path

Validate the metadata model against this future autonomous flow:

mctl_get_incident(...)
  → workflow_id + trace_id
  → locate failed span
  → derive time window / service / pod
  → query Loki
  → feed relevant logs/evidence into investigator

Actual autonomous trace/Loki querying by an agent can be a follow-up; this issue establishes the correlation primitives and exposes the identifiers needed to make it possible.

Privacy / reliability

  • Correlation must use IDs/metadata, not raw prompts/completions.
  • Logging/tracing backend failures must not block incident or workflow processing.
  • Avoid high-cardinality Loki labels for values like trace_id; keep them as structured fields unless the existing Loki setup explicitly supports/indexes them safely.

Acceptance criteria

  • Representative mctl-agent logs contain trace_id/span_id when emitted inside an active span.
  • Representative mctl-agents workflow logs use the agreed workflow/run correlation fields through fix(api): log every Telegram callback press #195/feat(scoutspike): add the spike scaffold to drafts that omit it #196.
  • A workflow execution can be followed to its trace and from there to matching Loki logs using documented identifiers.
  • Incident/evidence metadata includes trace_id when a trace exists.
  • At least one mctl operational/MCP response exposes the relevant trace/workflow identity where practical.
  • Correlation still degrades gracefully if the trace backend or Loki is unavailable.
  • A concrete failed workflow/incident example demonstrates incident → workflow → failed span → related logs.

Non-goals

  • Do not duplicate producer instrumentation already tracked in mctlhq/mctl-agent#38 or mctlhq/mctl-agents#195.
  • Do not reimplement execution identity/propagation owned by mctlhq/mctl-agents#196.
  • Do not introduce a Traceway-specific correlation contract; identifiers and fields must remain OpenTelemetry/backend neutral.

Activity

  1. mashkovd
    added a parent issue on Sep 10, 2026
  2. github-project-automation
    added this to mctl.ai — Roadmapon Sep 10, 2026
  3. mashkovd
    moved this to Todo in mctl.ai — Roadmapon Sep 10, 2026
  4. mashkovd
    marked this as blocked by 3 issues on Sep 23, 2026
  5. mashkovd
    added a commit that references this issue on Oct 11, 2026

    fix(observability): correlate the pipeline's failure-path log lines

    e302209

  6. mashkovd commented on Oct 12, 2026

    @mashkovd
    mashkovd
    on Oct 12, 2026
    MemberAuthor

    Evidence: organic failed incident → workflow → failed span → Loki (2026-10-12)

    Recorded before the Tempo eval teardown (2026-10-24), because the trace store is temporary. Nothing here was manufactured. Both failures below happened organically on 2026-10-11, inside DevLoop runs that started after tracing was enabled.

    Chain (AC 7)

    Step Identity How it was resolved
    Incident argo-mctl-agents-implement-dae24112-1791756910 (mctl-api, source=argo-workflows, type=workflow_failed, created 2026-10-11T22:15:10Z) mctl_list_incidents status=resolved
    Workflow mctl-agents-implement-dae24112, phase Failed (child … assert-produced failed), 22:07:49Z → 22:15:16Z; DevLoop dev-loop-mctlhq-mctl-telegram-757, run 01a12cf8-b7cb-700e-9d73-bf8f8aeab593 (started 21:57:39Z) named in the incident id and summary
    Trace 431486061c652efb376a860d9d656f25 (80 spans) mctl_get_workflow_status mctl-agents-implement-dae24112 → live.trace.trace_id (source: traceparent-argument), and audit.parameters.traceparent = 00-431486061c652efb376a860d9d656f25-7a16c732dc164e9e-01
    Failed spans argo.workflow mctl-agents-implement 7a16c732dc164e9e (error.type=Failed) → implementer.run 7a5cad2094fd54b9 (exit_1) → invoke_agent implementer 26c53dbda605f5f4 (ImplementerOrphanedSubagent, 22:08:14Z) → execute_tool Bash 8acf9997a91f7956 (tool_error). Fallback attempt: implementer.run f2658bdce26b9a69 → invoke_agent c27bb3b0b4b9d2c5 (http_403) → chat span 350b0c9a5933fc6f (oauth_org_not_allowed) Tempo eval, GET /api/traces/431486061c652efb376a860d9d656f25 through the Grafana datasource proxy (uid tempo-eval)
    Loki see below Grafana Loki datasource (uid P8E80F9AEF21F6940)

    The github.api.read exit_1 spans in this trace (5cc14bc48f2b1983, c23067abaea129e7) are the expected branch-existence 404 probe, as in #454. They are not used as the failure here.

    Loki lines, same service, pod and time window

    Primary implementer pod, {namespace="argo-workflows", pod="mctl-agents-implement-dae24112-run-implementer-3726176774"}, 22:07–22:15Z:

    22:08:11Z [trace] trace_id=431486061c652efb376a860d9d656f25 span_id=7a5cad2094fd54b9 argo_workflow=mctl-agents-implement-dae24112
22:13:28Z   fail mctl-telegram/issue-757-tool-availability-sli-counts-caller-erro: orphaned sub-agent: sub-agent task(s) never reported a terminal status within 300s: 1 task(s) still live: a904ce5754f57a456
22:13:29Z implementer exited 1

    The anchor's span_id is exactly the implementer.run span that parents the ImplementerOrphanedSubagent span.

    Fallback pod, pod="mctl-agents-implement-dae24112-run-implementer-1588491580", 22:13–22:16Z:

    22:14:06Z [trace] trace_id=431486061c652efb376a860d9d656f25 span_id=f2658bdce26b9a69 argo_workflow=mctl-agents-implement-dae24112
22:14:12Z AssistantMessage(... 'Your organization has disabled Claude subscription access for Claude Code ...', error='oauth_org_not_al...
22:14:13Z   fail mctl-telegram/issue-757-tool-availability-sli-counts-caller-erro: implementer produced no commits

    Worker side, {namespace=~"argo-workflows|admins"} |= "431486061c652efb376a860d9d656f25":

    22:07:49Z admins-mctl-agents-worker-implement-base-service-69c455c67szhgj  POST /api/v1/operations/mctl-agents-implement/execute 202 trace_id=431486061c652efb376a860d9d656f25 span_id=7a16c732dc164e9e

    Execution evidence carrying the same trace (mctl-api execution_evidence.trace_id): ev-ac179ebd1ff4d73a, ev-78de6bcaf3fcde7d, ev-5cf567cbf8c92bab.

    A second organic instance has the same shape: incident argo-mctl-agents-implement-af43e7b8-1791747930 → trace 008ad8979d34c0509e74d8062b7f92ca (ImplementerOrphanedSubagent, then fallback oauth_org_not_allowed).

    Other checks

    • mctl-agent producer live. mctl-agent:2.5.0 with three OTEL_* env vars since 2026-10-11T21:03:11Z. Two organic tickets since then, both traced, both escalated (no error span):
      • 1ac833c4-5ed7-4c10-9471-6c2aa0a28af5 → ac6c3b7f212ec9ac443529ddacc45078 (3 spans);
      • 6ebe8c19-7b63-40ad-ace4-1a8bfc0b56f8 → 3b54b8b36a8a3334173dc4cad2458e83 (4 spans, diagnose cpu_throttle).
      • Their log lines carry trace_id/span_id as JSON fields, and tickets.trace_id reads back from the store for both.
    • Evidence admin read. ev-68a4a54f06903df7: the column trace_id and the envelope execution.trace_id are both 164ac97d8d7264326c406240d263dcc1 (runtime ex-681b48cc77279413).
    • Privacy. All 19 real-producer traces were scanned for secret, prompt and body shapes: 0 hits. Attributes are metadata only, and the longest value is 47 characters.

    AC status

    AC Status
    mctl-agent logs carry trace_id/span_id met, live
    mctl-agents workflow logs use the agreed fields met, live (#195)
    Workflow → trace → Loki, documented met (docs/observability/correlation.md), live above
    Incident/evidence metadata includes trace_id partial. Execution evidence and the mctl-agent ticket store carry it. The mctl-api incident does not: mctl_get_incident on the incident above returns no trace_id, so the documented investigator path mctl_get_incident → trace_id does not work yet. Follow-up to be filed.
    An MCP response exposes trace identity met (mctl_get_workflow_status)
    Degrades gracefully met in code and tests (#167); no outage was induced
    Concrete failed incident example met: the chain above (owner accepted dae24112 for this AC on 2026-10-12)

    Remaining before close: the mctl_get_incident trace_id follow-up.

  7. mashkovd commented on Oct 12, 2026

    @mashkovd
    mashkovd
    on Oct 12, 2026
    MemberAuthor

    Follow-up for the remaining AC (incident metadata carries trace_id): mctlhq/mctl-api#585. Once it is live and mctl_get_incident argo-mctl-agents-implement-dae24112-1791756910 returns 431486061c652efb376a860d9d656f25, this issue has no open acceptance gap.

  8. mashkovd commented on Oct 13, 2026

    @mashkovd
    mashkovd
    on Oct 13, 2026
    MemberAuthor

    Closing on the owner's decision.

    • AC7 was accepted on 2026-10-12 with the organic DevLoop failure mctl-agents-implement-dae24112. The evidence is in comment 6130090541: incident, then ERROR spans, then the Loki anchor.
    • The last gap, mctl_get_incident returning the trace id, was fixed by mctl-api#585, released in mctl-api 4.82.0. It was re-verified live on 4.82.2: mctl_get_incident argo-mctl-agents-implement-dae24112-1791756910 returns trace_id: 431486061c652efb376a860d9d656f25 with trace_source: workflow_audit.
    • Related hardening: mctl-api#586 (failed audit reads are no longer treated as "no record") shipped in 4.82.1, and its review follow-ups shipped in 4.82.2.

    The Tempo evaluation (gitops#1280) is tracked separately and stays open until 2026-10-23.

  9. mashkovd
    closed this as completedon Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions