[mashkovd]

Part of mctlhq/.github#57 — step 2 of 2 of the observe rollout, and the one that starts
the soak.

Draft on purpose. Merge only after the bootstrap has been applied.

Order, and why this is last

1. feat(lifecycle): turn the ownership writer on, and make the bootstrap's apply reachable #1250 — the worker flips, the writer goes live
2. bootstrap dry_run=true, report reviewed row by row
3. bootstrap dry_run=false
4. this PR — the shepherd flips, the comparison starts against a populated store
5. the burst drains; then the soak clock starts

Starting the comparison before steps 1–3 measures the migration rather than the contract:
every entity the old mechanism already drives reads store-permits-old-forbids, the one
dangerous class, and the first week of data is the import.

What observe means

The store is read alongside the old DevLoop liveness probe, the two answers are compared
and logged, and the old one still decides. A test in mctl-agents asserts the filter's
decisions and the tick's own output are identical with the compare on and off.

This is the only rollout control — not workflow.patched("lifecycle-ownership"), a
history marker that can never be one, and not LIFECYCLE_OWNERSHIP_REQUIRED, the
break-glass inside enforce. Each is read in exactly one place.

To stop the soak: set this back to off. That is the remedy
LifecycleShadowDangerousDivergence names.

LifecycleShadowSilent ships here

It fires on the absence of the comparison counters, and until this commit that absence
is correct behaviour. It is the alert for the failure that reads as a clean soak.

It requires all three counters absent, not just agree: a compare emitting only
store-unknown is broken — a 403 on the admin-gated batch route answers that for every
id — and it is loud, with LifecycleShadowStoreUnknownHigh already on it. Keying on
agree alone would raise both alerts for one fault, naming different causes.

Expect a burst; do not start the clock on it

store-permits-old-forbids is non-zero by construction after the flip even with the
bootstrap applied. Executions started after the shepherd-in-loop patch but before
lifecycle-ownership never write a row — patched() replays False forever — and they stay
Running for up to the 14-day merge deadline.

Depends on

mctl-agents#378 merged, released, and the shepherd CWFT's agent_image on that release.
The mode means nothing on an image whose shepherd has no compare in it — which is exactly
what LifecycleShadowSilent would then report.

Verification

scripts/check-vm-rules.sh                          # check AND test, 0 failures
kubectl exec … env | grep LIFECYCLE_ROLLOUT_MODE   # on a shepherd step pod

Then: lifecycle_shadow_*_total appear with namespace="argo-workflows", no exported_
prefix, and store-unknown is not the whole volume.

🤖 Generated with Claude Code

https://claude.ai/code/session_01RUQTbQvUQjwbDFd7Nq9bvC