Skip to content

Gate soak is vacuous: dwell/sample count runs from repos that are not executing the candidate — and dev-lead's 'next' tier is structurally unoccupiable (SC2 forbids its only member) #1086

Open
0 of 3 issues completed

Description

@don-petry

For dev-lead, the next ring can never have an occupant. Its dwell and sample are therefore permanently vacuous, and the next->ring0 transition reports healthy soak numbers for a tier nobody executes.

The contradiction

standards/canary-rings.json defines dev-lead's first ring as the host repo alone:

"rings": [ { "channel": "next", "members": ["$host"], "order": 0 }, ... ]

$host is petry-projects/.github-private.

But petry-projects/.github-private's own dev-lead.yml is required by Safe Release SC2 to pin a stable-tier channel, asserted by tests/test_sc2_self_review_channel.bats (#1624, the automatable half of #503 / epic #495):

a deliberately-broken in-development version of this repo's agentic review/merge duty MUST NOT be able to block the PR that fixes its own breakage. The structural property that guarantees this … is that dev-lead.yml — although this repo sits in ring next — pins its dev-lead reusable to a stable-tier channel.

The test comment even acknowledges the tension ("although this repo sits in ring next") and resolves it in favour of stability — correctly. But that means next's only member is forbidden from pinning next. No other repo is a member. So the tier has no possible occupant, now or ever.

Why it matters

The gate measures a candidate's health across ring member repos. If a tier's members do not execute the candidate, the resulting dwell/sample carry no information about it — but they are reported as though they do.

Observed today on the ring1->stable frontier:

ring1->stable  BLOCKED  dwell=88h/12h  sample=196/1  cum_fail=3

196 executed runs against a floor of 1, and 88 hours of dwell against 12 — while no repo in the fleet was running the candidate:

.github              pins v1-ring0     -> 6d362318   (not the candidate)
.github-private      pins v139-stable  -> 40f6c232   (not the candidate)
markets              pins v139-stable  -> 40f6c232   (not the candidate)
TalkTerm             pins v1-ring1     -> 6d362318   (not the candidate)
bmad-bgreat-suite    pins v1-ring1     -> 6d362318   (not the candidate)
google-app-scripts   pins v139-stable  -> 40f6c232   (not the candidate)
ContentTwin          pins v139-stable  -> 40f6c232   (not the candidate)
                     candidate = 90dcc01c            (occupants: none)

All three counted failures came from repos running 40f6c232 or 6d362318. One of them was an exit-124 timeout, which the registry's workload-timeout suspect class asks "did this candidate change execution-time-sensitive code?" — a question that run cannot answer, because it never ran the candidate.

Separately from the empty-tier situation (which the repin work in #1063 fixes), the next tier's vacancy is structural and will survive that work.

What this is not

Options

  1. Make next explicitly unoccupied for self-hosting agents and skip it. Let ring0 be the first evidence-bearing tier for dev-lead, and have the gate report next->ring0 as NO_COVERAGE rather than a satisfied soak.
  2. Give next a real member — a non-self-hosting repo willing to run the bleeding edge. Note pr-review-trigger.yml already does exactly this for pr-review (pins pr-review/v1-next deliberately, and is explicitly excluded from the SC2 stable duty), so the pattern exists.
  3. Report coverage honestly. Whatever the topology, the gate should distinguish "N runs executed the candidate" from "N runs happened in member repos". This is the generally-correct fix and subsumes the others.

Acceptance criteria

  • The gate distinguishes runs that executed the candidate from runs that merely occurred in a member repo, and the sample figure counts only the former.
  • A transition whose members do not execute the candidate reports a distinct state (e.g. NO_COVERAGE) rather than a satisfied dwell/sample. It must not read as a clean soak.
  • cum_fail excludes failures from runs that did not execute the candidate — they cannot be evidence about it either way. (Note: this is not the same as the withdrawn Follow-up to #1047 (slice 2): exclude concurrency-evicted runs from cum_fail and sample, and name the counted run ids in evaluate output #1054; that one wrongly proposed counting cancelled runs. This is about version attribution.)
  • Resolve dev-lead's next tier explicitly — either give it a member, or mark it non-evidence-bearing — so the contradiction with SC2 is recorded in the registry rather than implicit.
  • Regression test using today's real topology: candidate 90dcc01c with zero occupants must not produce dwell=88h/12h sample=196/1.

Refs #1049, #1066, #1063, petry-projects/.github-private#1592, petry-projects/.github-private#1624, #503, #548, #657


Split 2026-09-13: now a tracking parent

The single pass timed out at the 2100s action budget (run 34735591102). That timed-out run then became the exact false positive this issue describes: it executed v139-ring0, yet it was counted as triage=REGRESSION against candidate ae4afa59, which no repo runs.

Do not implement from this parent.

Activity

  1. added
    bugBug reports
    dev-leadFor dev-lead agent pickup
    in-progressAn agent is actively working this issue
    on Sep 7, 2026
  2. don-petry commented on Sep 13, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead Implementation Plan

    Issue: #1086 — Gate soak is vacuous: dwell/sample count runs from repos that are not executing the candidate — and dev-lead's 'next' tier is structurally unoccupiable (SC2 forbids its only member)

    Scope

    Make the canary gate attribute soak evidence to repos that actually execute the candidate (occupancy), report a distinct NO_COVERAGE state when a transition's members do not run the candidate, exclude non-executing repos from sample/cum_fail, and record dev-lead's structurally-vacant next tier explicitly in the registry.

    Root cause

    The gate reasons about repos by ring membership but repos execute whatever tag they pin. _tier_sample / _cumulative_health count runs by date on member repos without checking whether the repo's pinned reusable resolves to the candidate commit. dev-lead's next ring has one member ($host = .github-private) which SC2 (test_sc2_self_review_channel.bats) requires to pin a stable channel — so next can never have an occupant.

    Implementation Checklist

    • scripts/lib/canary-rollout.sh — add pure decide_coverage <evidence_bearing> <positively_excluded> <occupant_count> → NO_COVERAGE|OK; update module header.
    • scripts/canary-rollout.sh — add _ring_field (read per-ring registry fields, e.g. evidence_bearing); add memoized _repo_executes_candidate <agent> <repo> <cand> (resolve the repo's caller-stub pin to a commit on the host and compare to the candidate) → yes|no|unknown; add _tier_occupancy (occupant/excluded counts + occupant repo list).
    • scripts/canary-rollout.sh _frontier_state — restrict sample to occupant source-tier repos, restrict _cumulative_health to occupant repos across all tiers, and emit NO_COVERAGE via decide_coverage (honouring evidence_bearing:false) instead of a satisfied dwell/sample.
    • scripts/canary-rollout.sh — surface NO_COVERAGE in cmd_evaluate and the fleet-status legend; ensure promote/promote-all never advance it and sync-issues does not open a blocker for it.
    • standards/canary-rings.json — mark dev-lead's next ring "evidence_bearing": false with a note recording the SC2 contradiction; extend registry description/member_tokens note.

    Tests to Write (tests/canary_rollout.bats)

    • decide_coverage: evidence_bearing=0 → NO_COVERAGE; occupants>0 → OK; occupants=0 + excluded>0 → NO_COVERAGE; all-indeterminate → OK (no false stall).
    • _repo_executes_candidate: pin resolves to candidate → yes; to a different commit → no; unreadable stub/tag → unknown (gh stubbed).
    • _tier_occupancy: mixed occupant/non-occupant members counted correctly.
    • _frontier_state regression (AC Add stacked PR strategy and Epic-level workflow guidance #5): dev-lead candidate 90dcc01c with zero occupants must report NO_COVERAGE with sample=0 — NOT dwell=88h/12h sample=196/1.
    • _cumulative_health: failures in non-occupant repos excluded from cum_fail.

    Test Command

    node_modules/.bin/bats tests/canary_rollout.bats and bash .dev-lead/scripts/dev-lead-lint.sh

  3. don-petry commented on Sep 13, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead: cannot implement issue #1086 — needs human attention

    A stage timed out (exit 124) while implementing this issue. A timeout is treated as a first-class, non-retryable condition — retrying with the same budget would just time out again — so this escalates for human review on the first occurrence instead of wasting a same-budget retry.

    This hit the max action budget. Split this issue into smaller, independently-implementable pieces, or raise vars.ACTION_TIMEOUT_SEC, then re-apply the dev-lead label.

    Partial work was checkpoint-pushed to dev-lead/issue-1086-20260913-0329. The engine produced changes before the timeout; rather than lose them, they were committed with the checkpoint(dev-lead): INCOMPLETE — marker and pushed to that branch. This work is partial and unreviewed — no pull request was opened and no review threads were resolved, so it must not be merged as-is. Use it as a salvage point: continue the branch by hand, or re-apply dev-lead after splitting the issue or raising the budget.

  4. removed
    in-progressAn agent is actively working this issue
    dev-leadFor dev-lead agent pickup
    dev-lead:needs-humandev-lead could not complete this issue; needs human attention
    on Sep 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugBug reports

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions