canary gate: a v-scoped channel tag shadows the release tag in _gh_candidate_cut_date, so any cross-repo agent adopting v<M>-<tier> channels is permanently BLOCKED (indeterminate) — blocks #1592

[don-petry]

Blocks petry-projects/.github-private#1592 (D-1). Found by hitting it live while seeding the dev-lead/v139-* channel family, then rolled back. Nothing is currently broken — this is the landmine that must be defused before the v-scoped channels can be adopted for a cross-repo agent.

The bug

_gh_candidate_cut_date (scripts/canary-rollout.sh ~356) enumerates release tags with:

gh api "repos/$repo/git/matching-refs/tags/$agent/v"

That prefix matches every <agent>/v… ref — so it catches the channel tags <agent>/v<M>-<tier> as well as the immutable releases <agent>/v<M>.<m>.<p>. It then returns the first ref whose dereferenced object equals the candidate commit:

if [ "$csha" = "$commit" ]; then _to_z "$cdate"; return 0; fi

A channel tag is lightweight (type=commit), so the else branch sets csha="$obj"; cdate="". And <agent>/v<M>-<tier> always points at a real commit — that is its entire purpose.

So the moment a v-scoped next channel exists for a cross-repo agent, it points at exactly the candidate commit, matches first, and returns an empty cut date.

Ordering makes it deterministic, not intermittent

matching-refs returns refs sorted, and - (0x2D) sorts before . (0x2E), so v139-next always precedes v139.4.0:

62:refs/tags/dev-lead/v139-next     33451103a2cf   commit    <- matched, cdate=""
72:refs/tags/dev-lead/v139.4.0      (annotated)              <- never reached

Observed live

With dev-lead/v139-{next,ring0,ring1,stable} seeded (v139-next → 33451103a2cf, the same commit as release v139.4.0):

candidate (v139-next) = 33451103a2cf  cut=
next->ring0    BLOCKED   dwell=0h/0h  sample=0/0  cum_fail=0
::warning::state=BLOCKED (indeterminate) — could not resolve the candidate's cut date,
so the gate fails closed and holds (not a detected failure; cum_fail=0).

After deleting the four v139-* tags, the same command on the same candidate:

candidate (next) = 33451103a2cf  cut=2026-08-31T23:37:09Z

Same commit, same release tag — the only difference is the presence of the channel tag.

Why this is load-bearing

The gate fails closed on an unresolvable cut date, so the effect is a permanent BLOCKED (indeterminate) hold: an agent that adopts v-scoped channels can never be promoted again. Since #1592's fix requires exactly that adoption for dev-lead (7 live stubs), this silently converts the migration into a fleet-wide promotion freeze.

It is also the per-candidate cumulative window start (#548) — health is measured since the candidate's own cut — so an empty value degrades the whole reliability verdict, not just a display field.

Why it has not bitten yet

Only the cross-repo path is affected. candidate_cut_date branches on host != THIS_REPO:

• Cross-repo (dev-lead, host .github-private) → _gh_candidate_cut_date, the API path, which yields "" for a lightweight tag. Broken.
• Same-repo (apply-repo-settings, host .github) → local git for-each-ref with %(creatordate:iso-strict), which for a lightweight tag returns the commit date — non-empty, so it silently "works".

So apply-repo-settings, which now has v1-next/v1-ring0 seeded (#989), is unaffected today purely because it is same-repo. But its same-repo path is also subtly wrong: it reports the candidate's commit date as the cut date, not the release tag's tagger date. Those differ whenever a release is cut some time after the commit — e.g. v139.4.0 commit 22:55:46Z vs tagged 23:37:09Z, a 41-minute skew applied to a dwell window. Worth fixing in the same change.

Acceptance criteria

• _gh_candidate_cut_date considers only immutable release tags — <agent>/v<major>.<minor>.<patch> — and skips channel tags <agent>/v<M>-<tier> and any other non-release <agent>/v… ref.
• Same filter applied to the local git for-each-ref path in candidate_cut_date, so both paths agree.
• The local path prefers the annotated tag's tagger date and does not silently substitute a lightweight tag's commit date.
• An annotated release tag whose commit equals the candidate resolves correctly even when a lightweight channel tag points at the same commit and sorts earlier. This is the regression test — use the real shape: dev-lead/v139-next (lightweight → X) alongside dev-lead/v139.4.0 (annotated → X).
• Cover both the cross-repo (API) and same-repo (for-each-ref) paths.
• After the fix, re-seeding dev-lead/v139-* must yield a resolvable cut= — that is the exit criterion for unblocking #1592 D-1 step 2.

Reproduction

R=petry-projects/.github-private
# seed a v-scoped next channel at the same commit as the newest release
gh api --method POST repos/$R/git/refs -f ref=refs/tags/dev-lead/v139-next \
  -f sha=33451103a2cfa8f64449df9030e6e7628381b406
gh workflow run canary-rollout.yml -R petry-projects/.github -f command=evaluate -f agent=dev-lead
#   => candidate (v139-next) = 33451103a2cf  cut=          <-- empty
gh api --method DELETE repos/$R/git/refs/tags/dev-lead/v139-next
#   => candidate (next) = 33451103a2cf  cut=2026-08-31T23:37:09Z

Related: petry-projects/.github-private#1592, #989, #657 (major-scoped channels, F4), #548, #1049.

[don-petry]

Dev-Lead Implementation Plan

Issue: #1046 — canary gate: a v-scoped channel tag shadows the release tag in _gh_candidate_cut_date

Scope

Make both the cross-repo (gh api matching-refs) and same-repo (git for-each-ref) candidate-cut-date paths consider only immutable annotated release tags <agent>/v<M>.<m>.<p>, skipping v-scoped channel tags <agent>/v<M>-<tier> that shadow the release tag and yield an empty (fail-closed) cut date.

Implementation Checklist

• scripts/lib/canary-rollout.sh: add pure helper _is_release_tag_suffix (matches v<M>.<m>.<p>, rejects v<M>-<tier> and other non-release suffixes)
• scripts/canary-rollout.sh _gh_candidate_cut_date (API path): capture .ref, filter to release-tag suffixes only before matching the commit
• scripts/canary-rollout.sh candidate_cut_date (local for-each-ref path): add %(refname) to the format, filter to release-tag suffixes, and match on the annotated tag's dereferenced commit + %(creatordate) (tagger date) — never a lightweight tag's commit date

Tests to Write

• _is_release_tag_suffix pure unit tests (accepts v139.4.0; rejects v139-next, next, v139, empty)
• _gh_candidate_cut_date regression: lightweight dev-lead/v139-next (→X, sorts first) alongside annotated dev-lead/v139.4.0 (→X) resolves the release's tagger date, not empty
• _gh_candidate_cut_date: only a channel tag at the commit → empty (no release found)
• candidate_cut_date same-repo path: lightweight channel + annotated release at same commit resolves the tagger date
• Update _crossrepo_stub/_crossrepo_promote_stub for-each-ref fixtures to the new 4-field annotated-release shape

Test Command

bats tests/canary_rollout.bats (+ bash .dev-lead/scripts/dev-lead-lint.sh)

[don-petry]

Dev-Lead: Implementation Complete

Plan Execution

• scripts/lib/canary-rollout.sh: added pure _is_release_tag_suffix (accepts v<M>.<m>.<p>, rejects v<M>-<tier>, bare tiers, v139, 139.4.0, empty)
• _gh_candidate_cut_date (cross-repo API path): captures .ref, filters every matching-refs entry through _is_release_tag_suffix before it can match the candidate commit — a channel tag can no longer shadow the release
• candidate_cut_date (same-repo for-each-ref path): format now carries %(refname), refs are filtered to the release suffix, and only the annotated tag's dereferenced commit + %(creatordate) (its tagger date) is used — a lightweight ref's commit date is never substituted
• Updated _crossrepo_stub / _crossrepo_promote_stub for-each-ref fixtures to the new 4-field annotated-release shape
• Full bats suite + dev-lead-lint.sh + shellcheck green

Acceptance criteria

• _gh_candidate_cut_date considers only immutable release tags <agent>/v<M>.<m>.<p>; skips <agent>/v<M>-<tier> and any other non-release <agent>/v… ref
• Same filter applied to the local git for-each-ref path so both paths agree
• Local path prefers the annotated tag's tagger date; does not substitute a lightweight tag's commit date
• Regression test uses the real shape — dev-lead/v139-next (lightweight → 33451103a2cf…) alongside dev-lead/v139.4.0 (annotated → same commit); resolves even though the channel tag sorts earlier
• Both the cross-repo (API) and same-repo (for-each-ref) paths covered
• Re-seeding dev-lead/v139-* yields a resolvable cut= — live exit criterion for #1592 D-1 step 2 (requires a real workflow run against the host; verified in unit tests, not run here)

Test Results

_is_release_tag_suffix: accepts a strict vMAJOR.MINOR.PATCH suffix .......... ok
_is_release_tag_suffix: rejects a v-scoped channel tag suffix (#1046 shadow)  ok
_is_release_tag_suffix: rejects bare channel tags and other non-release refs  ok
_gh_candidate_cut_date (API): annotated release resolves even when a
  lightweight v-channel tag sorts earlier at the same commit (#1046) ......... ok
_gh_candidate_cut_date (API): only a v-channel tag at the commit → empty .... ok
candidate_cut_date (cross-repo): dev-lead routes to API path, resolves past
  a shadowing v-channel tag (#1046) ......................................... ok
candidate_cut_date (same-repo for-each-ref): annotated release resolves its
  TAGGER date past a lightweight v-channel tag at the same commit (#1046) .... ok

Full suite: 305 passing, 0 failing
dev-lead-lint.sh: all checks passed   |   shellcheck --severity=warning -x: clean

Files Changed

• scripts/lib/canary-rollout.sh: new pure predicate _is_release_tag_suffix
• scripts/canary-rollout.sh: _gh_candidate_cut_date and candidate_cut_date now filter to release tags only; local path uses the annotated tagger date
• tests/canary_rollout.bats: 7 new tests (pure predicate + both cut-date paths + the canary gate: a v-scoped channel tag shadows the release tag in _gh_candidate_cut_date, so any cross-repo agent adopting v<M>-<tier> channels is permanently BLOCKED (indeterminate) — blocks #1592 #1046 regression); updated 2 existing for-each-ref stubs to the annotated-release shape

Notes

• Fix is order-independent: the channel tag is skipped by name regardless of sort order, so the same-commit - < . ordering can no longer wedge the gate.
• The final git log -1 --format=%cI <commit> fallback (candidate has no release tag yet) is unchanged.
• The last acceptance box is a live re-seed verification against .github-private, which a unit-test run cannot perform; the behaviour it checks is covered by the API-path regression test.