Skip to content

docs(adr): amend #1280 soak to admit DevLoop producer traces - #1788

Merged
mashkovd merged 1 commit into
mainfrom
docs/1280-amend-soak-devloop-producer
Oct 10, 2026
Merged

mashkovd merged 1 commit into
mainfrom
docs/1280-amend-soak-devloop-producer

Conversation

@mashkovd

Copy link
Copy Markdown
Member

Refs #1280, mctlhq/mctl-agents#195. Owner gate. This changes a declared measurement constraint, recorded before any producer emits, as the ADR's soak declaration requires.

Why

The soak declared "real producer traffic: none", and any producer must be recorded as an amendment before it lands. mctl-agents#195 closes only on a real DevLoop trace, and #1280's own preconditions say final scoring must use correctly exported live telemetry.

What changes

A new ADR section, "Producer amendment: DevLoop traces", plus two pointers in the runbook. Window, synthetic volume, storage, retention and teardown date are unchanged.

  • Producer: the mctl-agents DevLoop only.
    • The workers carry MCTL_TRACE_WORKFLOW_TYPES=DevLoopWorkflow, so reconcile, incidents, sweep, poller and schedules record nothing.
    • The investigate and implement pods carry MCTL_TRACE_REQUIRE_PARENT=true, so manual or sweep submits stay inert.
    • mctl-agent, mctl-api and every other base-service release stay off.
  • Volume: 101 investigations in the 14 days to 2026-10-10, about 7 a day.
    • Upper estimate per run: 1k spans for an investigation, 2k for an implementation.
    • That gives about 21k spans a day, or about 300k for the rest of the window.
    • Declared cap: 500,000 producer spans.
  • Stop rule: any one of these turns the producer off with a one-line revert of the workers' otel.enabled:
    • the cap is reached;
    • tempo_discarded_spans_total increases;
    • either Tempo exporter alert fires;
    • Tempo's working set passes 80% of its 1Gi limit;
    • the leak check on the first trace finds a single hit.
  • Not an adoption of Tempo, not a wider tracing rollout, not a window extension. The teardown commit also removes the producer env.

tests/test_adr_rubric_frozen.py passes.

🤖 Generated with Claude Code

https://claude.ai/code/session_013zqBqP9sSbVTd1kXRLA6AU

The declared soak said real producer traffic: none, and requires any
producer to be recorded as an amendment before it lands. mctl-agents#195
can only close on a real DevLoop trace, and #1280's own preconditions say
final scoring must use live telemetry, so the DevLoop is admitted as the
one producer, scoped by workflow type and parent sampling, with a declared
cap (500,000 spans) and a stop rule. Window and teardown are unchanged.

Refs #1280, mctlhq/mctl-agents#195

Claude-Session: https://claude.ai/code/session_013zqBqP9sSbVTd1kXRLA6AU
Copilot AI balanced review requested due to automatic review settings October 10, 2026 19:12

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@github-actions

Copy link
Copy Markdown
Contributor

Antigravity (agy) review — informational

Model: gemini-3.8-flash-medium · commit b6fd136.

No significant issues found.

@claude

claude Bot commented Oct 10, 2026 •

Copy link
Copy Markdown

Claude finished @mashkovd's task in 1m 30s —— View job


Review complete

Verdict: APPROVE (no P1/P2 findings)

This is a docs-only change: a new "Producer amendment: DevLoop traces" section in docs/adr/0001-agent-execution-trace-backend.md plus two pointers in docs/runbooks/tracing-bake-off.md. No Helm values, ArgoCD specs, RBAC, or workflow templates touched.

Checked for correctness against actual repo config:

  • MCTL_TRACE_WORKFLOW_TYPES and MCTL_TRACE_REQUIRE_PARENT are real env vars wired in platform-gitops/helm-charts/base-service/templates/deployment.yaml / rollout.yaml / _helpers.tpl, consistent with the amendment's description.
  • tempo_discarded_spans_total / Tempo exporter alerts referenced in the stop rule match platform-gitops/infra-components/observability/vm-rules/tempo-eval-alerts.yaml.
  • The amendment is additive (inserted before the ## Decision section) and doesn't touch the frozen rubric table or `` marker that tests/test_adr_rubric_frozen.py checks — confirmed by inspection that the new content contains no `| ... | ... |` table rows that could collide with the rubric-weights regex.
  • Window, synthetic volume, storage, retention, and teardown date are unchanged as claimed; the new cap (500,000 producer spans) and stop rule are clearly declared before any producer traffic, matching the owner-gate requirement in the PR description.

Submitted: gh api .../pulls/1788/reviews with event=APPROVE, bound to commit b6fd136e.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Docs-only ADR amendment and runbook pointers; env vars (MCTL_TRACE_WORKFLOW_TYPES, MCTL_TRACE_REQUIRE_PARENT) and metrics referenced match existing chart/alert config, no manifest or logic changes.

@mashkovd
mashkovd merged commit 211fced into main Oct 10, 2026
22 checks passed
@mashkovd
mashkovd deleted the docs/1280-amend-soak-devloop-producer branch October 10, 2026 19:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants