Skip to content

ci-failure-analyst can never reach stable: its ring1 members pin v1-stable, so the ring1->stable gate has no sample to count (stable frozen on v1.0.0 since 2026-07-10) #1270

Open

Description

@don-petry

Problem

ci-failure-analyst/v1-stable has pointed at v1.0.0 (cc0bc61f, 2026-07-10) since the channel was created. next, ring0 and ring1 advance normally, and next is now at v1.46.2. The ring1->stable gate can never pass, because the repos registered as ring1 members do not run the ring1 channel.

Evidence (2026-10-09)

A read-only gate evaluation, Canary Rollout run 37984012105:

* v1-next   -> f6f2e384d6bf
  v1-ring0  -> f34a76500759
  v1-ring1  -> f34a76500759
  v1-stable -> cc0bc61f70dc
next->ring0    SOAKING   dwell=3h/4h    sample=2/3  cum_fail=0
ring1->stable  SOAKING   dwell=61h/12h  sample=0/1  cum_fail=0

The dwell is met five times over. The sample is 0 of 1.

standards/canary-rings.json lists petry-projects/TalkTerm and petry-projects/bmad-bgreat-suite as the ring1 members for ci-failure-analyst. Their caller stubs pin stable, not ring1:

Repo ci-failure-analyst.yml pins Runs since 2026-10-07
TalkTerm ci-failure-analyst/v1-stable 0
bmad-bgreat-suite ci-failure-analyst/v1-stable 43, all skipped
google-app-scripts ci-failure-analyst/stable (legacy tag, also cc0bc61f) 38, all skipped
markets a raw SHA (b5851027…) 21, all skipped

The same TalkTerm repo pins -ring1 for dev-lead, agent-shield, auto-rebase, dependency-audit and most other agents, so this looks like a stub that was never moved rather than a deliberate choice.

Why nothing flagged it

  • The gate reports SOAKING, not BLOCKED, so sync-issues never opens a blocker issue. A soak that cannot finish looks the same as one that is in progress.
  • Every promote-all log line for this agent reads as normal.

Impact

Every consumer on the stable channel runs the three-month-old v1.0.0 analyst. 46 minor releases have not reached them.

Acceptance criteria

  1. TalkTerm and bmad-bgreat-suite pin ci-failure-analyst/v1-ring1, matching the registry.
  2. The gate has a defined answer for an agent whose runs are mostly skipped. I did not check whether a skipped run counts toward sample. If it does not, sample_min: 1 on ring1->stable may still take a long time to meet for a failure-triggered agent; decide whether to waive it, as ring0->ring1 already does with waive_sample.
  3. sync-issues flags a stalled soak: dwell met several times over while the sample stays at zero.
  4. A check compares each ring member's actual pin with its registered ring, for every agent. On TalkTerm, add-to-project and pr-auto-review also pin stable; confirm whether those repos are registered as ring1 members for those agents.
  5. google-app-scripts moves off the legacy ci-failure-analyst/stable tag, and markets moves off the raw SHA pin, or each is recorded as intentional.

Related

Found while closing out the Opus 5.5 canary in combsmith/.github-private#1899. ci-failure-analyst does not use the engine's deep tier, so it is outside that story.

Activity

  1. don-petry commented on Oct 10, 2026

    @don-petry
    ContributorAuthor

    Two more findings from a promotion attempt on 2026-10-10.

    1. bmad-bgreat-suite's stub is on the wrong trigger as well as the wrong channel. Its ci-failure-analyst.yml triggers on workflow_run, while the reusable reads only github.event.check_run.* (the template, this repo's lock file, TalkTerm and google-app-scripts all trigger on check_run). Under workflow_run the check_run payload is null, so the "Resolve non-fork PR" step fails:

    jq: error (at <stdin>:1): Cannot iterate over null (null)
    ##[error]Process completed with exit code 5.
    

    Evidence: bmad-bgreat-suite run 38052385125 (2026-10-10 12:33Z). That repo has never had a successful analyst run: of its last 300, 294 were skipped, 5 cancelled and 1 failed. That step is byte-identical in v1.0.0, v1.44.0, v1.48.0 and main, so no release fixes it; the stub has to move to check_run, or the reusable has to handle workflow_run.

    2. The gate reports that failure as a candidate regression. The run above executed ci-failure-analyst/v1-stable at cc0bc61f (v1.0.0), the old release. A promote run 55 minutes later (Canary Rollout run 38055710744) printed:

    gate=BLOCKED (triage=REGRESSION) for 'stable' [ring1->stable] — candidate regression suspected; not promoting. Investigate + rollback, do not --override blindly.
    

    For next->ring0 the same run correctly said "1 failure(s) in runs still on the previous release — informational, not counted against candidate". The ring1->stable evaluation did not apply that rule to a run on the previous stable release.

    Suggested additions to the acceptance criteria: move bmad-bgreat-suite's stub to the check_run trigger, and make the ring1->stable evaluation exclude failures from runs that executed the outgoing release.


    Generated by Claude Code

  2. don-petry commented on Oct 10, 2026

    @don-petry
    ContributorAuthor

    Update 2026-10-10: the bmad-bgreat-suite half of acceptance criterion 1 is now with dev-lead. combsmith/bmad-bgreat-suite#479 (the Fleet Monitor issue that has flagged that stub since 2026-09-19) carries the root cause and the fix: move the stub to the check_run trigger and pin ci-failure-analyst/v1-ring1. TalkTerm's stub already uses check_run; its pin is unchanged pending a decision here.

    Channel state at 20:57Z: v1-ring0 and v1-ring1 are on v1.48.0, v1-next on v1.50.0, v1-stable still on v1.0.0. The ring1->stable gate still reports BLOCKED (triage=REGRESSION) for the reason given above.


    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugBug reports

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions