Repository navigation
Gate soak is vacuous: dwell/sample count runs from repos that are not executing the candidate — and dev-lead's 'next' tier is structurally unoccupiable (SC2 forbids its only member) #1086
Description
For dev-lead, the next ring can never have an occupant. Its dwell and sample are therefore permanently vacuous, and the next->ring0 transition reports healthy soak numbers for a tier nobody executes.
The contradiction
standards/canary-rings.json defines dev-lead's first ring as the host repo alone:
"rings": [ { "channel": "next", "members": ["$host"], "order": 0 }, ... ]$host is petry-projects/.github-private.
But petry-projects/.github-private's own dev-lead.yml is required by Safe Release SC2 to pin a stable-tier channel, asserted by tests/test_sc2_self_review_channel.bats (#1624, the automatable half of #503 / epic #495):
a deliberately-broken in-development version of this repo's agentic review/merge duty MUST NOT be able to block the PR that fixes its own breakage. The structural property that guarantees this … is that
dev-lead.yml— although this repo sits in ringnext— pins its dev-lead reusable to a stable-tier channel.
The test comment even acknowledges the tension ("although this repo sits in ring next") and resolves it in favour of stability — correctly. But that means next's only member is forbidden from pinning next. No other repo is a member. So the tier has no possible occupant, now or ever.
Why it matters
The gate measures a candidate's health across ring member repos. If a tier's members do not execute the candidate, the resulting dwell/sample carry no information about it — but they are reported as though they do.
Observed today on the ring1->stable frontier:
ring1->stable BLOCKED dwell=88h/12h sample=196/1 cum_fail=3
196 executed runs against a floor of 1, and 88 hours of dwell against 12 — while no repo in the fleet was running the candidate:
.github pins v1-ring0 -> 6d362318 (not the candidate)
.github-private pins v139-stable -> 40f6c232 (not the candidate)
markets pins v139-stable -> 40f6c232 (not the candidate)
TalkTerm pins v1-ring1 -> 6d362318 (not the candidate)
bmad-bgreat-suite pins v1-ring1 -> 6d362318 (not the candidate)
google-app-scripts pins v139-stable -> 40f6c232 (not the candidate)
ContentTwin pins v139-stable -> 40f6c232 (not the candidate)
candidate = 90dcc01c (occupants: none)
All three counted failures came from repos running 40f6c232 or 6d362318. One of them was an exit-124 timeout, which the registry's workload-timeout suspect class asks "did this candidate change execution-time-sensitive code?" — a question that run cannot answer, because it never ran the candidate.
Separately from the empty-tier situation (which the repin work in #1063 fixes), the next tier's vacancy is structural and will survive that work.
What this is not
- Not an argument to weaken SC2. Keeping the self-host repo on a stable tier is correct and was validated in practice — it caught a mis-repin of that stub to
nextonpetry-projects/.github-private#1653. - Not the same as ship-drift reports '0 agents merged-but-unshipped' while all 7 dev-lead stubs are 29-181 commits behind — it compares the 'next' channel, which no stub pins (the #1592 detector is blind to #1592) #1049 (ship-drift compares the
nextchannel to main, and no stub pinsnext). That is a reporting bug; this is a registry/topology contradiction. They share a root cause — reasoning about repos by ring membership while repos execute whatever tag they pin — but need different fixes.
Options
- Make
nextexplicitly unoccupied for self-hosting agents and skip it. Letring0be the first evidence-bearing tier for dev-lead, and have the gate reportnext->ring0asNO_COVERAGErather than a satisfied soak. - Give
nexta real member — a non-self-hosting repo willing to run the bleeding edge. Notepr-review-trigger.ymlalready does exactly this for pr-review (pinspr-review/v1-nextdeliberately, and is explicitly excluded from the SC2 stable duty), so the pattern exists. - Report coverage honestly. Whatever the topology, the gate should distinguish "N runs executed the candidate" from "N runs happened in member repos". This is the generally-correct fix and subsumes the others.
Acceptance criteria
- The gate distinguishes runs that executed the candidate from runs that merely occurred in a member repo, and the
samplefigure counts only the former. - A transition whose members do not execute the candidate reports a distinct state (e.g.
NO_COVERAGE) rather than a satisfied dwell/sample. It must not read as a clean soak. -
cum_failexcludes failures from runs that did not execute the candidate — they cannot be evidence about it either way. (Note: this is not the same as the withdrawn Follow-up to #1047 (slice 2): exclude concurrency-evicted runs from cum_fail and sample, and name the counted run ids in evaluate output #1054; that one wrongly proposed countingcancelledruns. This is about version attribution.) - Resolve dev-lead's
nexttier explicitly — either give it a member, or mark it non-evidence-bearing — so the contradiction with SC2 is recorded in the registry rather than implicit. - Regression test using today's real topology: candidate
90dcc01cwith zero occupants must not producedwell=88h/12h sample=196/1.
Refs #1049, #1066, #1063, petry-projects/.github-private#1592, petry-projects/.github-private#1624, #503, #548, #657
Split 2026-09-13: now a tracking parent
The single pass timed out at the 2100s action budget (run 34735591102). That timed-out run then became the exact false positive this issue describes: it executed v139-ring0, yet it was counted as triage=REGRESSION against candidate ae4afa59, which no repo runs.
- [#1086 slice 1] Gate counts a failure against a candidate only if the run executed it (referenced_workflows attribution) #1120 slice 1: count a failure only if the run executed the candidate (
referenced_workflowsattribution). AC3. Blocked by canary-rollout ring promotion can skip the v-scoped channel tag, breaking every stub deployed to that tier #1065 and canary-rollout evaluates only the newest candidate — every new cut cancels a pending ring1→stable human confirmation, so dev-lead stable is frozen at 09-07 #1118, which edit the same file. - [#1086 slice 2] Dwell/sample count only candidate-executing runs; report NO_COVERAGE when none #1121 slice 2: dwell and sample count only candidate-executing runs;
NO_COVERAGEstate. AC1 + AC2. Blocked by [#1086 slice 1] Gate counts a failure against a candidate only if the run executed it (referenced_workflows attribution) #1120. - [#1086 slice 3] Record dev-lead's next-tier SC2 contradiction explicitly in canary-rings.json #1122 slice 3: record dev-lead's
next-tier SC2 contradiction in the registry. AC4. Blocked by [#1086 slice 2] Dwell/sample count only candidate-executing runs; report NO_COVERAGE when none #1121. - Related: workload-timeout suspect class can never match: signature is failed-step names, dev-lead's timeout step is 'Run issue' #1123, the
workload-timeoutsuspect class that can never match, blocked by [#1086 slice 1] Gate counts a failure against a candidate only if the run executed it (referenced_workflows attribution) #1120.
Do not implement from this parent.
Activity
don-petry commentedon Sep 13, 2026
Dev-Lead Implementation Plan
Issue: #1086 — Gate soak is vacuous: dwell/sample count runs from repos that are not executing the candidate — and dev-lead's 'next' tier is structurally unoccupiable (SC2 forbids its only member)
Scope
Make the canary gate attribute soak evidence to repos that actually execute the candidate (occupancy), report a distinct
NO_COVERAGEstate when a transition's members do not run the candidate, exclude non-executing repos fromsample/cum_fail, and record dev-lead's structurally-vacantnexttier explicitly in the registry.Root cause
The gate reasons about repos by ring membership but repos execute whatever tag they pin.
_tier_sample/_cumulative_healthcount runs by date on member repos without checking whether the repo's pinned reusable resolves to the candidate commit. dev-lead'snextring has one member ($host=.github-private) which SC2 (test_sc2_self_review_channel.bats) requires to pin a stable channel — sonextcan never have an occupant.Implementation Checklist
scripts/lib/canary-rollout.sh— add puredecide_coverage <evidence_bearing> <positively_excluded> <occupant_count>→NO_COVERAGE|OK; update module header.scripts/canary-rollout.sh— add_ring_field(read per-ring registry fields, e.g.evidence_bearing); add memoized_repo_executes_candidate <agent> <repo> <cand>(resolve the repo's caller-stub pin to a commit on the host and compare to the candidate) →yes|no|unknown; add_tier_occupancy(occupant/excluded counts + occupant repo list).scripts/canary-rollout.sh_frontier_state— restrictsampleto occupant source-tier repos, restrict_cumulative_healthto occupant repos across all tiers, and emitNO_COVERAGEviadecide_coverage(honouringevidence_bearing:false) instead of a satisfied dwell/sample.scripts/canary-rollout.sh— surfaceNO_COVERAGEincmd_evaluateand the fleet-status legend; ensurepromote/promote-allnever advance it andsync-issuesdoes not open a blocker for it.standards/canary-rings.json— mark dev-lead'snextring"evidence_bearing": falsewith a note recording the SC2 contradiction; extend registry description/member_tokens note.Tests to Write (tests/canary_rollout.bats)
decide_coverage: evidence_bearing=0 → NO_COVERAGE; occupants>0 → OK; occupants=0 + excluded>0 → NO_COVERAGE; all-indeterminate → OK (no false stall)._repo_executes_candidate: pin resolves to candidate → yes; to a different commit → no; unreadable stub/tag → unknown (gh stubbed)._tier_occupancy: mixed occupant/non-occupant members counted correctly._frontier_stateregression (AC Add stacked PR strategy and Epic-level workflow guidance #5): dev-lead candidate90dcc01cwith zero occupants must reportNO_COVERAGEwithsample=0— NOTdwell=88h/12h sample=196/1._cumulative_health: failures in non-occupant repos excluded fromcum_fail.Test Command
node_modules/.bin/bats tests/canary_rollout.batsandbash .dev-lead/scripts/dev-lead-lint.shdon-petry commentedon Sep 13, 2026
Dev-Lead: cannot implement issue #1086 — needs human attention
A stage timed out (exit 124) while implementing this issue. A timeout is treated as a first-class, non-retryable condition — retrying with the same budget would just time out again — so this escalates for human review on the first occurrence instead of wasting a same-budget retry.
timeoutThis hit the max action budget. Split this issue into smaller, independently-implementable pieces, or raise
vars.ACTION_TIMEOUT_SEC, then re-apply thedev-leadlabel.