Skip to content

feat(lifecycle): turn the ownership writer on, and make the bootstrap's apply reachable - #1250

Merged
mashkovd merged 3 commits into
mainfrom
feat/lifecycle-observe-flip
Sep 14, 2026
Merged

mashkovd merged 3 commits into
mainfrom
feat/lifecycle-observe-flip

Conversation

@mashkovd

@mashkovd mashkovd commented Sep 14, 2026 •

Copy link
Copy Markdown
Member

Part of mctlhq/.github#57 — step 1 of 2 of the observe rollout, and the only one that
can land before the store is bootstrapped.

Rewritten. This PR used to flip both sides at once and carried
LifecycleShadowSilent with it. Review of the bootstrap (mctlhq/mctl-agents#380) turned
up why that order was wrong; the shepherd half and the alert move to a follow-up.

What

LIFECYCLE_ROLLOUT_MODE=observe on the mctl-agents worker. That worker runs
DevLoopWorkflow, the only writer.

Why it goes first

Below observe the ownership activity short-circuits before any HTTP call. That is also
why the bootstrap refuses to --apply below observe: a row it wrote for a
devloop-workflow owner would never be heartbeated, progressed or released by the
workflow it names — it would sit active until its liveness bound expired, held by an
owner that does not know it holds anything.

So the order is:

  1. this PR — the writer goes live;
  2. bootstrap dry_run=true, report reviewed row by row;
  3. bootstrap dry_run=false;
  4. the shepherd flips (follow-up PR) — the comparison starts against a populated store;
  5. the burst drains; then the soak clock starts.

The previous ordering had the comparison starting against an empty store, which meant the
first week measured the import rather than the contract.

What this does not do

Nothing reads and nothing compares. The shepherd is untouched, and the comparison is what
produces the soak's evidence. Merging this changes what is recorded, not what is
decided.

The gate is in activity code, not workflow code, so this changes no command the
workflow issues: a history recorded under one mode replays byte-for-byte under another. A
test in mctl-agents asserts LIFECYCLE_ROLLOUT_MODE never appears in dev_loop.py.

Depends on

mctl-agents#378 merged and released, with this worker on that image. The rollout mode
means nothing on an image whose activity does not read it.

Verification

kubectl exec -n admins <worker pod> -- env | grep LIFECYCLE_ROLLOUT_MODE

Then: ownership rows start appearing for PRs the DevLoop drives — mctl_get_lifecycle_ownership
returns a non-zero count where it returned 0.

🤖 Generated with Claude Code

https://claude.ai/code/session_01RUQTbQvUQjwbDFd7Nq9bvC

@mashkovd
mashkovd force-pushed the feat/lifecycle-observe-flip branch from ae6e237 to 2337233 Compare September 14, 2026 07:22
@mashkovd mashkovd changed the title feat(lifecycle): start the observe soak feat(lifecycle): turn the ownership writer on Sep 14, 2026
@mashkovd mashkovd changed the title feat(lifecycle): turn the ownership writer on feat(lifecycle): turn the ownership writer on, and make the bootstrap's apply reachable Sep 14, 2026
Step 1 of the observe rollout, and the only one that can land before the
store is bootstrapped.

This worker runs DevLoopWorkflow, the only writer. Below `observe` its
ownership activity short-circuits before any HTTP call, so nothing is
recorded -- which is also why the bootstrap refuses to --apply below
observe: a row it wrote for a devloop-workflow owner would never be
heartbeated, progressed or released by the workflow it names, and would
sit `active` until its liveness bound expired.

So the writer goes live first, then the bootstrap imports the pull
requests already in flight, then the shepherd flips and the comparison
starts against a populated store.

Nothing reads or compares yet: the shepherd is untouched, and the
comparison is what produces the soak's evidence. The gate is in activity
code rather than workflow code, so this changes no command the workflow
issues and a history recorded under one mode replays under another.

Refs: mctlhq/.github#57

Claude-Session: https://claude.ai/code/session_01RUQTbQvUQjwbDFd7Nq9bvC
The template described two runs and no ordering constraint. The bootstrap
now refuses --apply below observe -- a row written while the writer is
off is never heartbeated by the owner it names -- so the rollout is three
steps and this workflow is the middle one. It also gained --service and
--slug, which are worth using for a first apply: an entity the store
already holds is skipped, so a row written wrongly cannot be corrected by
re-running.

Claude-Session: https://claude.ai/code/session_01RUQTbQvUQjwbDFd7Nq9bvC
…plicit

bootstrap.py refuses --apply below LIFECYCLE_ROLLOUT_MODE=observe, and
nothing in this template set that variable -- so --apply would have
exited 2 in production, always. The gate was correct and unreachable.

It is now a workflow parameter, and being a parameter is the point. The
mode bootstrap.py reads is THIS pod's, while the hazard it guards is the
Temporal worker's, which this pod cannot see: the gate is an operator
ATTESTATION, not a verification. Naming it in the submission -- dry_run=false
AND rollout_mode=observe -- makes that a deliberate two-part statement
rather than something a default carried in unnoticed, and the value is
echoed into the report so what was claimed sits beside what it licensed.

Defaults to "off" so a submission that forgets it refuses rather than
writes.

Refs: mctlhq/.github#57

Claude-Session: https://claude.ai/code/session_01RUQTbQvUQjwbDFd7Nq9bvC
@mashkovd
mashkovd force-pushed the feat/lifecycle-observe-flip branch from c6304ae to 7209355 Compare September 14, 2026 13:32
@mashkovd
mashkovd marked this pull request as ready for review September 14, 2026 13:37
Copilot AI lite review requested due to automatic review settings September 14, 2026 13:37
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

@claude

claude Bot commented Sep 14, 2026 •

Copy link
Copy Markdown

Claude finished @mashkovd's task in 2m 24s —— View job


Review complete

  • Fetch pinned diff (2084d011...72093558)
  • Read full context of both changed files
  • Check Argo Workflow parameter wiring (rollout_mode → env var)
  • Check operational safety / rollout ordering claims
  • Post inline findings (none needed)
  • Submit verdict

Scope: wft-lifecycle-bootstrap.yaml adds a rollout_mode workflow parameter (default "off", correctly quoted to avoid YAML 1.1's off→boolean coercion) wired straight into LIFECYCLE_ROLLOUT_MODE via env, mirroring the existing WORKFLOW_DRY_RUN/dry_run pattern exactly — no shell interpolation, so it doesn't trip validate-shell-param-interpolation.py. mctl-agents-worker/values.yaml sets LIFECYCLE_ROLLOUT_MODE: observe as a plain env entry.

Checked and found consistent:

  • Both new values are safe-by-default ("off" for the bootstrap CWFT, meaning a submission that omits -p rollout_mode=observe still refuses to write at --apply).
  • The rollout ordering claim in the PR body (writer → bootstrap dry-run → bootstrap apply → shepherd flip) matches what's actually wired here: this diff only flips the worker's env var and makes the bootstrap CWFT's apply path require an explicit attestation param; it does not touch the shepherd CWFT or lifecycle-rollout-alerts.yaml, consistent with the PR body's "shepherd half and the alert move to a follow-up."
  • Cross-checked against platform-gitops/infra-components/observability/vm-rules/lifecycle-rollout-alerts.yaml's comment ("LifecycleShadowSilent ... lands in the same change that sets LIFECYCLE_ROLLOUT_MODE=observe") — that commitment is about the shepherd's future flip (the shepherd emits the lifecycle_shadow_* counters that alert consumes), not the worker env var this PR sets, so no inconsistency is introduced here.
  • No secrets, RBAC, or resource changes; nothing in the diff touches shell-interpolated script bodies (the new env vars only ever reach the container via env:, never string-substituted into the sh -c script).

No P1/P2/P3 findings. This is a small, well-scoped, safe-by-default wiring change consistent with the repo's conventions and the PR's stated rollout plan.

gh api repos/mctlhq/mctl-gitops/pulls/1250/reviews --method POST -f commit_id="720935587ee5f56b1f1753bcb4a3da4ccfc5d40f" -f event="APPROVE" -f body="No P1/P2 findings (0 P3). Good to merge."

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No P1/P2 findings (0 P3). Good to merge.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Restrict bootstrap apply to the exact observe rollout mode.

Pull request overview

Enables lifecycle ownership recording in observe mode and wires rollout attestation into the bootstrap workflow.

Changes:

  • Sets the mctl-agents worker to LIFECYCLE_ROLLOUT_MODE=observe.
  • Adds and forwards the bootstrap rollout_mode parameter.
File summaries
File Summary
platform-gitops/services/admins/mctl-agents-worker/values.yaml Enables observe-mode ownership writes.
platform-gitops/argo-workflows/cluster-templates/wft-lifecycle-bootstrap.yaml Adds rollout-mode configuration; apply is not restricted to exactly observe (moderate issue).
Review details

Suppressed comments (1)

platform-gitops/argo-workflows/cluster-templates/wft-lifecycle-bootstrap.yaml:109

  • The template documents rollout_mode=observe as the required apply attestation, but bootstrap.py only checks rollout.records_writes(), which is also true for enforce and only. Thus dry_run=false -p rollout_mode=enforce can write rows after the shepherd has begun consuming the store, bypassing the staged order this template describes and potentially racing the reader. Reject any apply mode other than exactly observe (preferably in bootstrap.py, with a matching template guard).
      - name: rollout_mode
        value: "off"
  • Files reviewed: 2/2 changed files
  • Comments generated: 0
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@github-actions

Copy link
Copy Markdown
Contributor

Antigravity (agy) review — informational

Model: gemini-3.8-flash-medium · commit 7209355.

No significant issues found.

@mashkovd

Copy link
Copy Markdown
Member Author

Copilot's point is right and is fixed in bootstrap.py (mctlhq/mctl-agents#380), which is where it preferred it.

--apply now requires exactly observe rather than rollout.records_writes(). The reasoning it gave is the better half of the argument and was not the one the gate was written for:

  • below observe the writer is dead, and a row written then is never heartbeated by the owner it names;
  • at enforce and above the shepherd is already consuming the store, so a bulk import races a live reader and bypasses the staged order this template describes.

Two hazards, one on each side, so the gate has to be narrow to close both. records_writes() only ever closed the first.

No matching guard in the template: bootstrap.py refuses with exit 2 and emits a report naming the mode, and a second copy of the rule here would be a second thing to keep in step. The template's text already says -p rollout_mode=observe, which is now exactly what is enforced.

Worth recording that the same review round found the deeper version of this: the parameter is an operator attestation, not a verification — it sets the mode in the bootstrap pod, while the hazard belongs to the Temporal worker, which that pod cannot see. That is stated in the template comment and in the module docstring, and the attested mode is echoed into the report.

@mashkovd
mashkovd merged commit eefd30b into main Sep 14, 2026
25 checks passed
@mashkovd
mashkovd deleted the feat/lifecycle-observe-flip branch September 14, 2026 14:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants