Repository navigation
ci-failure-analyst can never reach stable: its ring1 members pin v1-stable, so the ring1->stable gate has no sample to count (stable frozen on v1.0.0 since 2026-07-10) #1270
Description
Problem
ci-failure-analyst/v1-stable has pointed at v1.0.0 (cc0bc61f, 2026-07-10) since the channel was created. next, ring0 and ring1 advance normally, and next is now at v1.46.2. The ring1->stable gate can never pass, because the repos registered as ring1 members do not run the ring1 channel.
Evidence (2026-10-09)
A read-only gate evaluation, Canary Rollout run 37984012105:
* v1-next -> f6f2e384d6bf
v1-ring0 -> f34a76500759
v1-ring1 -> f34a76500759
v1-stable -> cc0bc61f70dc
next->ring0 SOAKING dwell=3h/4h sample=2/3 cum_fail=0
ring1->stable SOAKING dwell=61h/12h sample=0/1 cum_fail=0
The dwell is met five times over. The sample is 0 of 1.
standards/canary-rings.json lists petry-projects/TalkTerm and petry-projects/bmad-bgreat-suite as the ring1 members for ci-failure-analyst. Their caller stubs pin stable, not ring1:
| Repo | ci-failure-analyst.yml pins |
Runs since 2026-10-07 |
|---|---|---|
| TalkTerm | ci-failure-analyst/v1-stable |
0 |
| bmad-bgreat-suite | ci-failure-analyst/v1-stable |
43, all skipped |
| google-app-scripts | ci-failure-analyst/stable (legacy tag, also cc0bc61f) |
38, all skipped |
| markets | a raw SHA (b5851027…) |
21, all skipped |
The same TalkTerm repo pins -ring1 for dev-lead, agent-shield, auto-rebase, dependency-audit and most other agents, so this looks like a stub that was never moved rather than a deliberate choice.
Why nothing flagged it
- The gate reports SOAKING, not BLOCKED, so
sync-issuesnever opens a blocker issue. A soak that cannot finish looks the same as one that is in progress. - Every
promote-alllog line for this agent reads as normal.
Impact
Every consumer on the stable channel runs the three-month-old v1.0.0 analyst. 46 minor releases have not reached them.
Acceptance criteria
- TalkTerm and bmad-bgreat-suite pin
ci-failure-analyst/v1-ring1, matching the registry. - The gate has a defined answer for an agent whose runs are mostly skipped. I did not check whether a skipped run counts toward
sample. If it does not,sample_min: 1onring1->stablemay still take a long time to meet for a failure-triggered agent; decide whether to waive it, asring0->ring1already does withwaive_sample. sync-issuesflags a stalled soak: dwell met several times over while the sample stays at zero.- A check compares each ring member's actual pin with its registered ring, for every agent. On TalkTerm,
add-to-projectandpr-auto-reviewalso pin stable; confirm whether those repos are registered as ring1 members for those agents. - google-app-scripts moves off the legacy
ci-failure-analyst/stabletag, and markets moves off the raw SHA pin, or each is recorded as intentional.
Related
- "Gate soak is vacuous: dwell/sample count runs from repos that are not executing the candidate" (Gate soak is vacuous: dwell/sample count runs from repos that are not executing the candidate — and dev-lead's 'next' tier is structurally unoccupiable (SC2 forbids its only member) #1086) is the earlier form of this. Now that runs are attributed to the candidate, the same pin mismatch produces a deadlock instead of a vacuous pass.
- "Canary Rollout times out: 6 of the last 7 scheduled runs hit timeout-minutes: 30" (Canary Rollout times out: 6 of the last 7 scheduled runs hit timeout-minutes: 30 and were cancelled — rings stop advancing silently #1175) and "canary-rollout: gate evaluation ~40% slower since feat: implement issue #1224 — canary-rollout health gate goes blind on an ADR-0007 collapsed repo: it resolves runs by per-role workflow NAME, which the ingress replaces with jobs #1238" (canary-rollout: gate evaluation ~40% slower since #1238 — 5 of 9 complete scheduled runs now hit the 30-minute timeout #1259) are separate. 23 of the last 30 scheduled sweeps were cancelled at the timeout, which delays every agent, but fixing that would not unblock this one.
Found while closing out the Opus 5.5 canary in combsmith/.github-private#1899. ci-failure-analyst does not use the engine's deep tier, so it is outside that story.
Activity
don-petry commentedon Oct 10, 2026
Two more findings from a promotion attempt on 2026-10-10.
1. bmad-bgreat-suite's stub is on the wrong trigger as well as the wrong channel. Its
ci-failure-analyst.ymltriggers onworkflow_run, while the reusable reads onlygithub.event.check_run.*(the template, this repo's lock file, TalkTerm and google-app-scripts all trigger oncheck_run). Underworkflow_runthecheck_runpayload is null, so the "Resolve non-fork PR" step fails:Evidence: bmad-bgreat-suite run 38052385125 (2026-10-10 12:33Z). That repo has never had a successful analyst run: of its last 300, 294 were skipped, 5 cancelled and 1 failed. That step is byte-identical in
v1.0.0,v1.44.0,v1.48.0andmain, so no release fixes it; the stub has to move tocheck_run, or the reusable has to handleworkflow_run.2. The gate reports that failure as a candidate regression. The run above executed
ci-failure-analyst/v1-stableatcc0bc61f(v1.0.0), the old release. A promote run 55 minutes later (Canary Rollout run 38055710744) printed:For
next->ring0the same run correctly said "1 failure(s) in runs still on the previous release — informational, not counted against candidate". Thering1->stableevaluation did not apply that rule to a run on the previous stable release.Suggested additions to the acceptance criteria: move bmad-bgreat-suite's stub to the
check_runtrigger, and make thering1->stableevaluation exclude failures from runs that executed the outgoing release.Generated by Claude Code
don-petry commentedon Oct 10, 2026
Update 2026-10-10: the bmad-bgreat-suite half of acceptance criterion 1 is now with dev-lead. combsmith/bmad-bgreat-suite#479 (the Fleet Monitor issue that has flagged that stub since 2026-09-19) carries the root cause and the fix: move the stub to the
check_runtrigger and pinci-failure-analyst/v1-ring1. TalkTerm's stub already usescheck_run; its pin is unchanged pending a decision here.Channel state at 20:57Z:
v1-ring0andv1-ring1are onv1.48.0,v1-nextonv1.50.0,v1-stablestill onv1.0.0. Thering1->stablegate still reportsBLOCKED (triage=REGRESSION)for the reason given above.Generated by Claude Code