feat(incidents): store and return trace_id from mctl_get_incident #585

Problem

mctl_get_incident returns no trace_id, so the investigator path that mctlhq/mctl-agent#97 defines (mctl_get_incident(...) → workflow_id + trace_id → failed span → Loki) breaks at its first step. This is the last open acceptance gap on #97 ("Incident/evidence metadata includes trace_id when a trace exists"). The owner accepted the rest of #97 on 2026-10-12: see the evidence at mctlhq/mctl-agent#97 (comment).

Verified 2026-10-12:

• The alerts table (internal/alerts/store.go) has no trace column. No mctl-api table except execution_evidence and model_usage_records stores a trace_id.
• mctl-agent already sends it. Since mctl-agent 2.5.0 (chore(main): release 4.32.2 #167), internal/mctlclient/client.go sends trace_id on the incident publish and on each status PATCH. mctl-api decodes leniently and drops it. Example: ticket 6ebe8c19-7b63-40ad-ace4-1a8bfc0b56f8 has trace_id=3b54b8b36a8a3334173dc4cad2458e83 in the agent's store, but mctl_get_incident on that id returns none.
• Argo-sourced incidents (argo-<workflow>-<epoch>, posted by the onExit legs of cwft-mctl-agents-{implement,investigate,run,shepherd}) send no trace at all. For implement and investigate, the trace is already durable in mctl-api: mctl_get_workflow_status recovers it from the submit's audit record (audit.parameters.traceparent). Example: incident argo-mctl-agents-implement-dae24112-1791756910 → workflow mctl-agents-implement-dae24112 → traceparent=00-431486061c652efb376a860d9d656f25-….

Proposed minimal fix

1. Store and return. Add alerts.trace_id TEXT NOT NULL DEFAULT ''.
   • Accept trace_id on POST /api/v1/incidents and on the PATCH.
   • Validate it: 32 lowercase hex characters, not all zeros. Otherwise ignore it and leave the stored value unchanged, never fail the request. This keeps the "tracing failure never blocks incident processing" rule.
   • On PATCH, a later run's value replaces the stored one, matching the ticket store's "latest processing run" semantics.
   • Return it, omitted when empty, from GET /api/v1/incidents/{id}, the list endpoint and mctl_get_incident.
   • This alone makes every mctl-agent ticket carry its trace with no producer change.
2. Argo incidents.
   • Preferred: when an incident with source=argo-workflows has no stored trace_id, resolve it at read time from the audit record of the workflow named by the incident. That is the same source mctl_get_workflow_status already uses, so no producer changes are needed, and it covers historical incidents such as dae24112 immediately, which makes the fix verifiable without waiting for a new failure.
     • The workflow name should come from a structured field, not from parsing the id. Either add a workflow_name that the onExit legs send, or match the audit record by the workflow name in the summary URL as a fallback.
     • Mark a derived value as such in the response (for example trace_source: "workflow_audit").
   • Alternative: the onExit legs of implement and investigate send trace_id taken from {{workflow.parameters.traceparent}}. This only covers new incidents and needs a gitops change.
3. Graceful degradation.
   • An audit lookup that fails, or finds no record, returns the incident without trace_id and with an explicit trace_unavailable reason.
   • It must not be reported as "no trace". Per AGENTS.md, "could not observe" is never "observed absent".
4. Tests.
   • Store round trip.
   • An invalid trace_id is ignored, not rejected.
   • The read-time derivation for an argo incident, plus the audit-miss case.
   • A regression test that fails when the column or the response field is removed.

Acceptance

• mctl_get_incident 6ebe8c19-7b63-40ad-ace4-1a8bfc0b56f8 (or the next mctl-agent ticket after deploy) returns its trace_id.
• mctl_get_incident argo-mctl-agents-implement-dae24112-1791756910 returns trace_id=431486061c652efb376a860d9d656f25. That requires the preferred option in step 2.
• Processing an incident never fails because of trace_id.

Non-goals

• Not storing span_id on the incident (feat: add MCP trigger for mctl-agents incident responder #97: a span id names a transient attempt).
• Not querying Tempo or Loki from mctl-api.

Refs mctlhq/mctl-agent#97, mctlhq/.github#55