From 82c06cdbc028c3de1367e5253ec4d66a6e031b41 Mon Sep 17 00:00:00 2001 From: mashkovd Date: Mon, 14 Sep 2026 09:22:14 +0200 Subject: [PATCH 1/3] feat(lifecycle): turn the ownership writer on Step 1 of the observe rollout, and the only one that can land before the store is bootstrapped. This worker runs DevLoopWorkflow, the only writer. Below `observe` its ownership activity short-circuits before any HTTP call, so nothing is recorded -- which is also why the bootstrap refuses to --apply below observe: a row it wrote for a devloop-workflow owner would never be heartbeated, progressed or released by the workflow it names, and would sit `active` until its liveness bound expired. So the writer goes live first, then the bootstrap imports the pull requests already in flight, then the shepherd flips and the comparison starts against a populated store. Nothing reads or compares yet: the shepherd is untouched, and the comparison is what produces the soak's evidence. The gate is in activity code rather than workflow code, so this changes no command the workflow issues and a history recorded under one mode replays under another. Refs: mctlhq/.github#57 Claude-Session: https://claude.ai/code/session_01RUQTbQvUQjwbDFd7Nq9bvC --- .../admins/mctl-agents-worker/values.yaml | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/platform-gitops/services/admins/mctl-agents-worker/values.yaml b/platform-gitops/services/admins/mctl-agents-worker/values.yaml index 27b91fe49..7373c5bc6 100644 --- a/platform-gitops/services/admins/mctl-agents-worker/values.yaml +++ b/platform-gitops/services/admins/mctl-agents-worker/values.yaml @@ -89,6 +89,23 @@ env: TEMPORAL_ADDRESS: temporal-frontend.temporal.svc.cluster.local:7233 TEMPORAL_NAMESPACE: mctl-agents LOG_LEVEL: info + # The lifecycle rollout stage (ADR-010 §12, mctlhq/.github#57). + # + # This worker runs DevLoopWorkflow, the only WRITER. It flips FIRST, ahead of + # the shepherd CWFT and ahead of the ownership bootstrap, and the order is + # not cosmetic: below `observe` the ownership activity short-circuits before + # any HTTP call, so a row the bootstrap wrote for a devloop-workflow owner + # would never be heartbeated, progressed or released by the workflow it + # names. It would sit `active` until its liveness bound expired, held by an + # owner that does not know it holds anything. + # + # Writer live -> bootstrap -> shepherd flips and the comparison starts + # against a populated store. + # + # The gate is in ACTIVITY code, not workflow code, so this does not change + # which commands the workflow issues: a history recorded under one mode + # replays under another. + LIFECYCLE_ROLLOUT_MODE: observe # GitHub App installation tokens expire after 60 min. rotate-github-app-tokens # mints a fresh one every 30 min into Vault, but env vars are frozen at # container start, so a long-lived pod like this one is authenticated for at From 9b9355e6731767d1b714613399f48a3eef6c884e Mon Sep 17 00:00:00 2001 From: mashkovd Date: Mon, 14 Sep 2026 09:23:08 +0200 Subject: [PATCH 2/3] docs(lifecycle): say what the bootstrap refuses and when The template described two runs and no ordering constraint. The bootstrap now refuses --apply below observe -- a row written while the writer is off is never heartbeated by the owner it names -- so the rollout is three steps and this workflow is the middle one. It also gained --service and --slug, which are worth using for a first apply: an entity the store already holds is skipped, so a row written wrongly cannot be corrected by re-running. Claude-Session: https://claude.ai/code/session_01RUQTbQvUQjwbDFd7Nq9bvC --- .../wft-lifecycle-bootstrap.yaml | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/platform-gitops/argo-workflows/cluster-templates/wft-lifecycle-bootstrap.yaml b/platform-gitops/argo-workflows/cluster-templates/wft-lifecycle-bootstrap.yaml index 037f8fc1b..c9a3682e3 100644 --- a/platform-gitops/argo-workflows/cluster-templates/wft-lifecycle-bootstrap.yaml +++ b/platform-gitops/argo-workflows/cluster-templates/wft-lifecycle-bootstrap.yaml @@ -22,6 +22,23 @@ metadata: NOTHING. Review every row against the store and live Temporal. 2. dry_run=false — writes the rows the report planned. + A dry run works in any rollout mode. An APPLY refuses unless the worker + is already at LIFECYCLE_ROLLOUT_MODE=observe, and exits 2 saying so: + below observe the DevLoopWorkflow's ownership activity short-circuits, so + a row written here for a devloop-workflow owner would never be + heartbeated, progressed or released by the workflow it names. It would + sit `active` until its liveness bound expired, held by an owner that does + not know it holds anything. + + So the rollout is three steps and this is the middle one: + 1. the WORKER flips to observe — the writer goes live + 2. this workflow, dry then apply + 3. the SHEPHERD flips — the comparison starts against a populated store + + --service and --slug scope a run. Worth using for the first apply: + apply_report skips an entity the store already holds, so a row written + wrongly cannot be corrected by re-running. + Safety properties, all in orchestrator/lifecycle/bootstrap.py: - READ FIRST. Any entity that reads UNKNOWN aborts the whole run with nothing written: an id whose owner cannot be seen is not one From 720935587ee5f56b1f1753bcb4a3da4ccfc5d40f Mon Sep 17 00:00:00 2001 From: mashkovd Date: Mon, 14 Sep 2026 09:31:43 +0200 Subject: [PATCH 3/3] fix(lifecycle): make the bootstrap's apply reachable, and its gate explicit bootstrap.py refuses --apply below LIFECYCLE_ROLLOUT_MODE=observe, and nothing in this template set that variable -- so --apply would have exited 2 in production, always. The gate was correct and unreachable. It is now a workflow parameter, and being a parameter is the point. The mode bootstrap.py reads is THIS pod's, while the hazard it guards is the Temporal worker's, which this pod cannot see: the gate is an operator ATTESTATION, not a verification. Naming it in the submission -- dry_run=false AND rollout_mode=observe -- makes that a deliberate two-part statement rather than something a default carried in unnoticed, and the value is echoed into the report so what was claimed sits beside what it licensed. Defaults to "off" so a submission that forgets it refuses rather than writes. Refs: mctlhq/.github#57 Claude-Session: https://claude.ai/code/session_01RUQTbQvUQjwbDFd7Nq9bvC --- .../wft-lifecycle-bootstrap.yaml | 26 +++++++++++++++++-- 1 file changed, 24 insertions(+), 2 deletions(-) diff --git a/platform-gitops/argo-workflows/cluster-templates/wft-lifecycle-bootstrap.yaml b/platform-gitops/argo-workflows/cluster-templates/wft-lifecycle-bootstrap.yaml index c9a3682e3..4bffb79cd 100644 --- a/platform-gitops/argo-workflows/cluster-templates/wft-lifecycle-bootstrap.yaml +++ b/platform-gitops/argo-workflows/cluster-templates/wft-lifecycle-bootstrap.yaml @@ -22,8 +22,9 @@ metadata: NOTHING. Review every row against the store and live Temporal. 2. dry_run=false — writes the rows the report planned. - A dry run works in any rollout mode. An APPLY refuses unless the worker - is already at LIFECYCLE_ROLLOUT_MODE=observe, and exits 2 saying so: + A dry run works in any rollout mode. An APPLY additionally needs + `-p rollout_mode=observe` — the operator attesting that the mctl-agents + worker is already flipped. Without it bootstrap.py exits 2, because below observe the DevLoopWorkflow's ownership activity short-circuits, so a row written here for a devloop-workflow owner would never be heartbeated, progressed or released by the workflow it names. It would @@ -64,6 +65,9 @@ metadata: -p agent_image=ghcr.io/mctlhq/mctl-agents: \ -p dry_run=true + and, for the apply, additionally -p rollout_mode=observe. That is an + attestation and not a check: this pod cannot see the worker's mode. + agent_image has no default on purpose: the module does not exist in any released image yet, so there is no value that could work. Argo refuses the submission by name rather than failing inside the container on a @@ -101,6 +105,22 @@ spec: # The DEFAULT, and deliberately the safe one. bootstrap.py requires # --apply to write anything, so an operator who submits this template # with no parameters at all gets a report and no writes. + - name: rollout_mode + value: "off" + # The operator's ATTESTATION that the mctl-agents worker is already at + # LIFECYCLE_ROLLOUT_MODE=observe — not a verification of it. This + # parameter sets the variable in THIS pod, and the hazard it guards is + # the WORKER's mode, which this pod cannot see. + # + # Without it --apply is unreachable in production: bootstrap.py refuses + # to write below observe and exits 2, and nothing else in this template + # sets the variable. With it, an apply is a deliberate two-part + # statement — dry_run=false AND rollout_mode=observe — rather than + # something a default carried in unnoticed. + # + # Defaults to "off" so a submission that forgets it refuses rather than + # writes. The value is echoed into the report, so what was claimed is + # on the record beside what it licensed. - name: agent_image # NO DEFAULT, deliberately, and unlike every sibling template. # @@ -214,6 +234,8 @@ spec: env: - name: WORKFLOW_DRY_RUN value: "{{workflow.parameters.dry_run}}" + - name: LIFECYCLE_ROLLOUT_MODE + value: "{{workflow.parameters.rollout_mode}}" - name: STATE_DIR value: /workdir/mctl-gitops/platform-gitops/agents-state # The ownership store is admin-gated: GET /lifecycle/ownership/batch