Repository navigation
canary #668 increment 6: SUSPECT→PRE_EXISTING auto-downgrade (decide_suspect_downgrade) — closes #668 #688
Description
Increment 6 of #668 (optional 1c of the design) — data-driven SUSPECT→PRE_EXISTING auto-downgrade. Closes out the #668 series. Additive, default-off, opt-in per suspect class; every agent/class without the new knob behaves exactly as increment 2 shipped. Keep scripts/lib/canary-rollout.sh pure; the new bats are now CI-gated by Canary-rollout Tests (#685).
Rationale
Increment 2 gave the ambiguous middle a SUSPECT verdict (BLOCKS + needs-human + guidance) — right when a human must judge "is this timeout the candidate's fault?". But when the candidate's suspect-class failure rate is no worse than the prior version's on comparable traffic, that judgment is mechanical: it's environmental (PRE_EXISTING, report-only), not a candidate regression. Auto-answering the statistically clear cases removes toil; the genuinely ambiguous ones still stop for a human.
The gate already computes per-day baselines (_baseline_daily) and the SUSPECT machinery (_suspect_patterns, classify_failure, increment 2). This extends classification to compare rates over the baseline window.
Scope
1. Pure core (scripts/lib/canary-rollout.sh)
Add a side-effect-free decide_suspect_downgrade <cand_rate_permille> <base_rate_permille> <base_sample> <knobs_json> → DOWNGRADE | HOLD:
HOLD(stay SUSPECT) whenbase_sample < min_baseline_sample— never downgrade on thin data (the tiny-n guard the design calls out; conservative = keep the human).DOWNGRADEwhencand_rate_permille <= base_rate_permille + margin_permille(candidate no worse than baseline for this class → environmental).- else
HOLD.
Knobs object:{min_baseline_sample, margin_permille}. Keepclassify_failure's existing 3-arg signature back-compatible; the downgrade is applied by the orchestrator AFTER classify_failure returns SUSPECT (SUSPECT + DOWNGRADE → PRE_EXISTING), so the pure verdict core stays small and unit-testable.
2. Orchestrator (scripts/canary-rollout.sh)
Only when triage == SUSPECT and the matched suspect class carries auto_downgrade:
- Compute the candidate's suspect-class failure rate (matched-failures / executed) over the in-window candidate runs, and the same class's rate over the trailing baseline window (prior version) — reuse
_baseline_daily+ the suspect-match signature machinery; no new API surface beyond what SUSPECT already fetches. - Feed both rates + baseline sample to
decide_suspect_downgrade.DOWNGRADE→ override the verdict to PRE_EXISTING (report-only, advances with--allow-pre-existing, no needs-human);HOLD→ stay SUSPECT (unchanged). - Emit a
::notice::recording the rate comparison and the decision.
3. Registry (standards/canary-rings.json, jq -a to preserve em-dashes)
Per-suspect-class opt-in — extend dev-lead's workload-timeout class (increment 2) with:
"auto_downgrade": { "min_baseline_sample": 10, "margin_permille": 100 }Classes without auto_downgrade never downgrade (stay SUSPECT). Start with dev-lead workload-timeout only.
4. sync-issues
A downgraded (SUSPECT→PRE_EXISTING) failure files/updates a PRE_EXISTING blocker (report-only, no needs-human), and the body shows the rate comparison (candidate vs baseline permille, sample sizes) + notes it auto-cleared the SUSPECT hold on statistical grounds. A held SUSPECT is unchanged (increment 2 behaviour).
5. Tests (tests/canary_rollout.bats)
decide_suspect_downgrade: DOWNGRADE (cand ≤ base+margin, sample≥min); HOLD (cand > base+margin); HOLD (base_sample < min — tiny-n guard); boundary at exactly base+margin.- orchestrator: SUSPECT + auto_downgrade + cand-rate-not-worse → PRE_EXISTING (report-only, no needs-human); SUSPECT + auto_downgrade + cand-rate-worse → stays SUSPECT+needs-human; SUSPECT + auto_downgrade + thin baseline → stays SUSPECT.
- class WITHOUT auto_downgrade → SUSPECT unchanged (increment 2 regression guard).
Out of scope
- Applying auto_downgrade to other classes/agents (dev-lead workload-timeout only initially).
- Any change to REGRESSION or the reliability gate.
Acceptance
lib/pure;decide_suspect_downgradeunit-tested incl. the tiny-n guard.- Only
auto_downgrade-flagged SUSPECT classes can downgrade, and only when statistically no-worse-than-baseline on sufficient data; everything else stays SUSPECT+needs-human (increment 2 byte-identical for un-flagged classes). - bats green under
Canary-rollout Tests; shellcheck clean. - Closes the Enhancement: canary gate should assess correctness, not just run-reliability #668 increment series (Layers 0–3 + 1c all delivered).
Activity
don-petry commentedon Jul 12, 2026
Dev-Lead Implementation Plan
Issue: #688 — canary #668 increment 6: SUSPECT→PRE_EXISTING auto-downgrade (decide_suspect_downgrade) — closes #668
Scope
Add a default-off, opt-in-per-suspect-class data-driven SUSPECT→PRE_EXISTING auto-downgrade: when a candidate's suspect-class failure rate is statistically no-worse-than the prior version's over the baseline window (and the baseline isn't too thin), the mechanical "environmental, not a regression" judgment is auto-made; genuinely ambiguous/worse cases still stop for a human.
Implementation Checklist
scripts/lib/canary-rollout.sh— add puredecide_suspect_downgrade <cand_permille> <base_permille> <base_sample> <knobs_json>→DOWNGRADE|HOLD(tiny-n guard first, then cand ≤ base+margin, else HOLD)scripts/canary-rollout.sh— add_suspect_downgrade_patterns(auto_downgrade-flagged classes only) +_suspect_class_counts(matched-failures/executed over a window)scripts/canary-rollout.sh— in_frontier_state, when triage==SUSPECT (failure-class, not correctness-shift) and the matched class opts in, compare rates and override SUSPECT→PRE_EXISTING on DOWNGRADE; emit a::notice::; plumb rate fields on the state linescripts/canary-rollout.sh— update the 3_frontier_stateread sites +_blocker_body(render rate comparison + auto-clear note for a downgraded PRE_EXISTING; no needs-human)standards/canary-rings.json— addauto_downgrade {min_baseline_sample:10, margin_permille:100}to dev-lead'sworkload-timeoutclass onlyTests to Write (TDD, before implementation)
decide_suspect_downgrade: DOWNGRADE (cand ≤ base+margin, sample≥min); HOLD (cand worse); HOLD (tiny-n baseline); boundary exactly at base+marginTest Command
bats tests/canary_rollout.bats(plusbash .dev-lead/scripts/dev-lead-lint.shandshellcheck)feat: implement issue #688 — canary #668 increment 6: SUSPECT→PRE_EXI…