Repository navigation
canary-rollout ring promotion can skip the v-scoped channel tag, breaking every stub deployed to that tier #1065
Description
Summary
When canary-rollout promotes an agent into a ring, it sometimes moves only the bare channel tag <agent>/<tier> and never creates the major-scoped <agent>/v<M>-<tier>. The canary itself keeps working — _resolved_channel prefers the v-scoped tag and falls back to the bare one — so the gap is invisible until a stub deploy needs the v-scoped ref, at which point it pins a nonexistent tag and the consumer fails at startup.
Evidence
Sweep of every bare channel tag on petry-projects/.github against its v-scoped counterpart:
| Agent | Has | Missing |
|---|---|---|
apply-repo-settings |
ring1 |
v1-ring1 — created by hand 2026-09-02 to unblock #1045 |
persona-mention |
ring0, ring1 |
v1-ring0, v1-ring1 — still missing |
Every other agent has both forms at every tier it has reached (add-to-project/v1-ring1, agent-shield/v2-ring1, auto-rebase/v2-ring1, dependency-audit/v2-ring1, pr-auto-review/v1-ring1, …).
persona-mention has v1-next and releases v1.0.0–v1.0.3, so a major exists to scope against — this is not a "no major yet" case.
For apply-repo-settings the inconsistency is sharp and one hop wide:
v1-next edbac2551e4c
v1-ring0 edbac2551e4c ← next→ring0 created the v-scoped form
ring1 edbac2551e4c ← ring0→ring1 did NOT
Same agent, same candidate, consecutive promotions.
Why it stays hidden
_resolved_channel (epic #657 Phase F4) prefers <agent>/v<M>-<tier> and falls back to the bare tier when it is absent, so evaluation, promotion and rollback all keep functioning on a bare-only tag.
The consumer side does not fall back. deploy-standard-workflows.sh keys the stub pin on the channel major — the highest existing <base>/v<M>-<tier> tag — and its own comment states the rule:
Once a channel tag exists, a bare stub is drift and must migrate to the tier's
v<M>-<tier>form.
So for any agent that has any v-scoped channel tag, the deploy emits v<M>-<tier> for the target repo's tier. If the promotion never created that ref, every stub deployed to that tier pins a tag that does not exist — a fleet-wide startup_failure for that workflow.
This is what blocked #1045: TalkTerm and bmad-bgreat-suite resolve to tier=ring1, so the stub pins @apply-repo-settings/v1-ring1, which the ring0→ring1 promotion had not created.
AC1 — promotion must move both forms, or neither
A ring promotion that moves <agent>/<tier> must also create/move <agent>/v<M>-<tier> whenever the agent has a channel major. Treat the pair as one atomic operation so the two can never diverge.
AC2 — backfill persona-mention
Create persona-mention/v1-ring0 and persona-mention/v1-ring1 at the commits their bare tags point to. Until then, any stub deploy targeting a ring0/ring1 repo for persona-mention will pin a nonexistent ref.
AC3 — a drift check
canary-rollout drift (the read-only registry audit) should report any bare channel tag lacking its v-scoped counterpart for an agent that has a channel major. This sweep found the problem in one pass and should not have needed a human to run it:
gh api repos/petry-projects/.github/git/refs/tags --jq '.[].ref|sub("refs/tags/";"")' > tags.txt
for tier in next ring0 ring1 stable; do
for bare in $(grep -E "/${tier}$" tags.txt); do
agent="${bare%/*}"
grep -qE "^${agent}/v[0-9]+-${tier}$" tags.txt || echo "$agent — has /$tier but NO v<M>-$tier"
done
doneAC4 — regression coverage
A test asserting that promoting an agent which has a channel major produces both the bare and the v-scoped tag at the target tier.
Note on the manual tag. apply-repo-settings/v1-ring1 was created by hand at edbac2551e4c091d5ee0c86195f9a34682c2bfe1 (matching ring1) to unblock the #1045 rollout. It is a lightweight ref, consistent with how ring1 and v1-ring0 are stored. It should be adopted by the automation once AC1 lands, not treated as a special case.
Filed from a manual compliance sweep, 2026-09-02.
CORRECTION 2026-09-07 — AC1 above is wrong. Replaced by AC1′.
AC1 as originally written ("promotion must move both forms, or neither") misreads the design. The promotion is meant to move exactly one tag. From canary-rollout.sh:
# Move the RESOLVED frontier tag (major-scoped-channels epic #657, F4): advance the
# v-scoped `<agent>/v<M>-<tier>` within its major line when it exists, else the legacy
# bare `<agent>/<tier>`. A v2 promotion never touches a v1 tag.
local frontier_tag; frontier_tag="$(_resolved_channel_tag "$agent" "$frontier")"and _resolved_channel prefers the v-scoped tag only when it already resolves to a real oid, else falls back to the bare tier. Moving both would break the F4 contract — a bare-only legacy agent must not sprout v-scoped tags, and a v2 line must never touch v1.
AC1′ — the real defect is bootstrap, not duplication
Once an agent has a channel major (any <agent>/v<M>-<tier> tag exists anywhere), the v-scoped line can never extend to a tier it has not already reached:
- Promotion to tier T calls
_resolved_channel "$agent" "$T". <agent>/v<M>-Tdoes not exist yet → its commit is empty →_looks_like_oidfails.- Resolution falls back to the bare
<agent>/Tand moves that. <agent>/v<M>-Tis therefore never created, and step 2 fails identically on every future promotion.
The v-scoped line is stranded at whatever tier it was first seeded at. This is exactly what happened to apply-repo-settings:
v1-next edbac2551e4c ← seeded
v1-ring0 edbac2551e4c ← seeded
ring1 edbac2551e4c ← ring0→ring1 fell back to the bare tag
(v1-ring1 never created; added by hand 2026-09-02)
Required behaviour: when _agent_current_major returns a major for the agent, promotion to tier T must create <agent>/v<M>-T at the candidate rather than falling back to the bare tier. The fallback stays correct for agents with no channel major at all (the legacy bare-only fleet), so the condition is "has a channel major", not "the tag already exists".
Note _gh_move_tag already creates a missing ref (PATCH, then POST on a genuine 404/422), so the primitive supports this — only the resolution step needs to stop falling back when a major exists.
AC2 — unchanged, and now more urgent
Backfill persona-mention/v1-ring0 and persona-mention/v1-ring1 at the commits their bare tags point to. persona-mention has v1-next and releases v1.0.0–v1.0.3, so it has a channel major and is squarely in the AC1′ trap.
Urgency is raised by #1088: once RING_REUSABLES is derived from the registry, persona-mention becomes ring-aware in deploy-standard-workflows.sh, and emit_ref_for will compute @persona-mention/v1-ring0 / v1-ring1 for repos at those tiers — refs that do not exist. #1088's own AC2 (refuse to deploy an unresolvable ref) turns that into a loud failure rather than a broken fleet, but the tags still need to exist for the deploy to succeed.
Current gaps, verified 2026-09-07 — apply-repo-settings no longer appears because its v1-ring1 was created by hand:
persona-mention — has /ring0, no v<M>-ring0
persona-mention — has /ring1, no v<M>-ring1
AC4′ — coverage, restated for AC1′
Replace the original AC4. Assert that promoting an agent which has a channel major into a tier whose v-scoped tag does not yet exist creates <agent>/v<M>-<tier> and does not move the bare <agent>/<tier>. Pair it with the inverse: an agent with no channel major still moves the bare tier tag and creates no v-scoped tag.
AC3 — unchanged
The drift sweep remains as written and would have caught both agents in one pass.
SCOPE FENCE 2026-09-08 — released to dev-lead for the CODE fix only
In scope for this issue
AC1′, AC3, AC4′ — the resolution/bootstrap fix in scripts/canary-rollout.sh, the drift check, and coverage.
OUT of scope — do not attempt
AC2 (backfill persona-mention/v1-ring0 and persona-mention/v1-ring1) is a maintainer action. It requires creating release channel tags, which an agent must not do. Leave it entirely alone; it is tracked here for the maintainer and will be handled separately.
More generally, for this issue: do not create, move, delete or cut any tag or release. The fix is to the code that decides which tag to move — not to the tags themselves. A correct implementation changes only scripts/canary-rollout.sh (and/or scripts/lib/ring-pins.sh) plus tests.
Verification without touching tags
Prove AC1′ with the bats suite, not by running a promotion. A promotion against the live host would move real channel tags and reposition agents in the canary. tests/canary_rollout.bats already stubs gh; extend that pattern.
Related work in flight
- RING_REUSABLES has drifted from canary-rings.json — 7 agents deploy verbatim, 2 with pins that do not exist #1088 (merged) —
RING_REUSABLESnow equals the registry, and the deploy refuses a non-resolving pin. That refusal is what currently turns thepersona-mentiongap into a loud error rather than a broken fleet. - ring_tier_for_repo is agent-agnostic but canary ring membership is per-agent — derive it from the registry #1092 —
ring_tier_for_repoderived from the registry. Same file family; rebase on currentmainand coordinate if both are in flight.
Why AC1′ matters beyond persona-mention
Every agent that later gains a channel major hits the same bootstrap trap the first time it is promoted into a tier its v-scoped line has not reached. apply-repo-settings hit it at ring0→ring1 and needed a hand-cut tag; persona-mention is stuck at two tiers. Without AC1′ this recurs silently for the next agent, and it is only ever noticed when a stub deploy fails.
Maintainer review notes on PR #1097 (added to the BODY, 2026-09-10 — dev-lead cannot read issue comments, #1566)
Both verified against the PR head 7ce32b16, not inferred. Neither blocks the approach — the fix
is right and _resolved_channel's removal of the bare-tier fallback for established-major agents
does make every unreached v-scoped tier promotable, including ring1→stable. These are two defects
to clear before merge.
1. channel_commit's docstring now contradicts the function (scripts/canary-rollout.sh:341-346)
The comment still reads:
Resolution is fall-back-safe on the major dimension (epic #657 F4): prefer the v-scoped tag, else the bare tier.
That is precisely the behaviour this PR removes. _resolved_channel (line 323) now returns the
v-scoped tag with an empty commit when the tag is absent, and only falls back to the bare tier for
an agent with no channel major at all. The next reader debugging a resolution will be misled by
a comment sitting directly above the call that contradicts it. Update it to describe the
established-major / legacy-bare-only split.
2. AC3's drift report cannot see the agent AC3 exists to catch (scripts/canary-rollout.sh:2530)
[ -n "$(_channel_tag_commit "$agent" "$tier")" ] || continue # no bare tier tag → nothing to pairA tier with no bare tag is skipped, so a gap is only visible when the bare side exists. Live
state today:
| agent | bare tags | v-scoped tags | missing | seen by drift? |
|---|---|---|---|---|
apply-repo-settings |
next, ring0, ring1, stable |
v1-next, v1-ring0, v1-ring1 |
v1-stable |
yes |
persona-mention |
ring0, ring1 |
v1-next, v1-ring0, v1-ring1 |
v1-stable, and bare next/stable |
no |
persona-mention has no bare next and no bare stable, so its missing v1-stable is invisible
to the report — and persona-mention is the one agent this issue's own Evidence table names as
"still missing". The AC3 report will read clean for it while the promoter (correctly) creates the
tag underneath.
A tier can drift by missing either side of the pair. Suggested AC refinement: for an agent with
an established channel major, report a tier where either <agent>/<tier> or
<agent>/v<M>-<tier> is absent while the other is present — or, more simply, report any tier the
agent has reached whose v-scoped form does not exist, independent of the bare form.
Not a defect, recorded so it is not re-litigated
_gh_move_tag's 422-vs-404 misclassification is fixed on main — the matcher accepts
"reference does not exist" alongside not found / http 404. It is not a contributing cause here.
Status 2026-09-13: core fix merged in PR #1097; the remaining scope is exactly two items
PR #1097 merged at 782e005f (2026-09-13T04:03Z). It delivered bare-tier fallback removal for established-major agents, _agent_current_channel_major with all three call sites switched, the release-major docstring on _agent_current_major, and the divergent release/channel test. Do not redo any of that. The PR carried no closing reference, which is why this issue is still open.
Remaining; implement only this. Both are verified present on main at 782e005f:
- Stale docstring,
scripts/canary-rollout.sh:364-365.channel_commitstill says "Resolution is fall-back-safe on the major dimension (epic Story (F): major-version boundary in the channel/ring model [#604] #657 F4): prefer the v-scoped tag, else the bare tier." Since feat: implement issue #1065 — canary-rollout ring promotion can skip the v-scoped channel tag, breaking every stub deployed to that tier #1097, that is false for any agent with a channel major. Rewrite it to describe the established-major vs legacy-bare-only split. - AC3 drift blind spot,
scripts/canary-rollout.sh:2555._channel_tag_major_gapsskips any tier with no bare tag (# no bare tier tag → nothing to pair), sopersona-mention, which has no barenextorstable, reports clean whilev1-stableis missing. Report a tier the agent has reached whosev<M>-<tier>form is absent regardless of whether the bare form exists. Test: an agent withv1-next,v1-ring0,ring0, no barestable, nov1-stable, placed in a ring layout where stable is reached, is flagged.
Close this issue when those two land.
Activity
don-petry commentedon Sep 8, 2026
Dev-Lead Implementation Plan
Issue: #1065 — canary-rollout ring promotion can skip the v-scoped channel tag (AC1′/AC3/AC4′, code fix only)
Scope
Fix the resolution/bootstrap trap in
scripts/canary-rollout.shso that once an agent has anestablished channel major (a
<agent>/v<M>-nextanchor tag), channel resolution stops fallingback to the bare tier — letting a promotion create
<agent>/v<M>-<tier>instead of strandingthe v-line — plus an AC3 drift check and AC4′ regression coverage. No tags are created/moved
(AC2 backfill is a maintainer action, per the scope fence).
Implementation Checklist
scripts/canary-rollout.sh: add_agent_has_channel_major <agent> <major>(true iff thev<M>-nextanchor exists) — the single "has a channel major" signal.scripts/canary-rollout.sh: rework_resolved_channelso that when the agent has a channelmajor it resolves the v-scoped
<agent>/v<M>-<tier>(possibly-empty commit) and does NOTfall back to the bare tier; the legacy bare-only fleet (no channel major) is unchanged.
scripts/canary-rollout.sh(cmd_drift): add_channel_tag_major_gaps <agent>+ aDRIFT[channel-tag]section reporting any bare tier tag lacking itsv<M>-<tier>counterpartfor an agent that has a channel major (AC3).
Tests to Write (
tests/canary_rollout.bats)creates
v<M>-<tier>and never moves the bare tier tag._channel_tag_major_gapsflags a bare-only tier; reports nothing for a bare-only fleet.canary-rollout driftsurfaces aDRIFT[channel-tag]finding.Test Command
bats tests/canary_rollout.bats(plusbash .dev-lead/scripts/dev-lead-lint.sh)don-petry commentedon Sep 8, 2026
Dev-Lead: Progress (in progress — not yet durable)
Plan Execution (intended)
<agent>/v<M>-nextanchor exists) resolves the v-scoped tier tag and no longer falls backto the bare tier — a promotion into a not-yet-reached tier now creates
<agent>/v<M>-<tier>(AC1′). The legacy bare-only fleet (a release major but no v-line) is unchanged.
canary-rollout driftnow reports any bare tier tag lacking itsv<M>-<tier>counterpart for an agent with a channel major.Files touched (this workspace, not yet pushed)
scripts/canary-rollout.sh: new_agent_has_channel_major; reworked_resolved_channel;new
_channel_tag_major_gaps+ aDRIFT[channel-tag]section incmd_drift.tests/canary_rollout.bats: AC4′ (bootstrap + inverse) promote tests, AC3 gap-helper unittests, and an AC3
driftintegration test.Local check output (provisional — not the pushed run)
bats tests/canary_rollout.batsandbash .dev-lead/scripts/dev-lead-lint.shwere runlocally in this workspace; the durable results come from the post-push run.
Notes
v<M>-nextchannel anchor,not merely a
vX.Y.Zrelease — this preserves the F4 transition-safety contract for thebare-only fleet while fixing the bootstrap trap for agents like persona-mention.
don-petry commentedon Sep 8, 2026
Dev-Lead: Implementation Complete — PR #1097
The implementation for issue #1065 is durable: commits are pushed and a pull request is open. This record is backed by verifiable artifacts you can confirm in one click.
73b86f8a1f899a1b040ad985d73577a901bbc5bcReview happens on the PR. Acceptance criteria and test results are evidenced by the PR's diff and CI checks — not asserted here. If a later stage fails, this claim is retracted in place.
don-petry commentedon Sep 8, 2026
AC2 DONE — persona-mention backfilled 2026-09-08
Both cut as lightweight commit refs at the SHAs their bare counterparts already pointed to. Verified matching. AC2 is closed; AC1′/AC3/AC4′ remain (PR #1097).
Refinement for AC3 — the drift check needs a registry filter
Running the AC3 sweep across both hosts (not just
petry-projects/.github) surfaces one remaining hit, and it is a false positive:pr-reviewis not instandards/canary-rings.json— it is manually managed onnext/stableonly. Its barering0andring1are vestigial: both point atded84ce4820d, identical tonext, untouched since 2026-06-21, and never independently advanced. Its real channels are consistent —next/v1-nextbothded84ce4820d, andstable(an annotated tag) andv1-stableboth resolve to commitceab48a1c64d.Cutting
pr-review/v1-ring0/v1-ring1would manufacture two more vestigial refs and fix nothing. Do not.AC3 amended: the drift check must consider only agents present in
standards/canary-rings.json, and should sweep every host inorg_infra_reposrather than assuming.github. Without the registry filter it reports unregistered, manually-managed agents as drift on every run — the class of false positive that has already cost this fleet real dispatches (.github-private#1571/#1674).feat: implement issue #1065 — canary-rollout ring promotion can skip …