Skip to content

canary #668 increment 6: SUSPECT→PRE_EXISTING auto-downgrade (decide_suspect_downgrade) — closes #668 #688

Closed

Description

@don-petry

Increment 6 of #668 (optional 1c of the design) — data-driven SUSPECT→PRE_EXISTING auto-downgrade. Closes out the #668 series. Additive, default-off, opt-in per suspect class; every agent/class without the new knob behaves exactly as increment 2 shipped. Keep scripts/lib/canary-rollout.sh pure; the new bats are now CI-gated by Canary-rollout Tests (#685).

Rationale

Increment 2 gave the ambiguous middle a SUSPECT verdict (BLOCKS + needs-human + guidance) — right when a human must judge "is this timeout the candidate's fault?". But when the candidate's suspect-class failure rate is no worse than the prior version's on comparable traffic, that judgment is mechanical: it's environmental (PRE_EXISTING, report-only), not a candidate regression. Auto-answering the statistically clear cases removes toil; the genuinely ambiguous ones still stop for a human.

The gate already computes per-day baselines (_baseline_daily) and the SUSPECT machinery (_suspect_patterns, classify_failure, increment 2). This extends classification to compare rates over the baseline window.

Scope

1. Pure core (scripts/lib/canary-rollout.sh)

Add a side-effect-free decide_suspect_downgrade <cand_rate_permille> <base_rate_permille> <base_sample> <knobs_json> → DOWNGRADE | HOLD:

  • HOLD (stay SUSPECT) when base_sample < min_baseline_sample — never downgrade on thin data (the tiny-n guard the design calls out; conservative = keep the human).
  • DOWNGRADE when cand_rate_permille <= base_rate_permille + margin_permille (candidate no worse than baseline for this class → environmental).
  • else HOLD.
    Knobs object: {min_baseline_sample, margin_permille}. Keep classify_failure's existing 3-arg signature back-compatible; the downgrade is applied by the orchestrator AFTER classify_failure returns SUSPECT (SUSPECT + DOWNGRADE → PRE_EXISTING), so the pure verdict core stays small and unit-testable.

2. Orchestrator (scripts/canary-rollout.sh)

Only when triage == SUSPECT and the matched suspect class carries auto_downgrade:

  • Compute the candidate's suspect-class failure rate (matched-failures / executed) over the in-window candidate runs, and the same class's rate over the trailing baseline window (prior version) — reuse _baseline_daily + the suspect-match signature machinery; no new API surface beyond what SUSPECT already fetches.
  • Feed both rates + baseline sample to decide_suspect_downgrade. DOWNGRADE → override the verdict to PRE_EXISTING (report-only, advances with --allow-pre-existing, no needs-human); HOLD → stay SUSPECT (unchanged).
  • Emit a ::notice:: recording the rate comparison and the decision.

3. Registry (standards/canary-rings.json, jq -a to preserve em-dashes)

Per-suspect-class opt-in — extend dev-lead's workload-timeout class (increment 2) with:

"auto_downgrade": { "min_baseline_sample": 10, "margin_permille": 100 }

Classes without auto_downgrade never downgrade (stay SUSPECT). Start with dev-lead workload-timeout only.

4. sync-issues

A downgraded (SUSPECT→PRE_EXISTING) failure files/updates a PRE_EXISTING blocker (report-only, no needs-human), and the body shows the rate comparison (candidate vs baseline permille, sample sizes) + notes it auto-cleared the SUSPECT hold on statistical grounds. A held SUSPECT is unchanged (increment 2 behaviour).

5. Tests (tests/canary_rollout.bats)

  • decide_suspect_downgrade: DOWNGRADE (cand ≤ base+margin, sample≥min); HOLD (cand > base+margin); HOLD (base_sample < min — tiny-n guard); boundary at exactly base+margin.
  • orchestrator: SUSPECT + auto_downgrade + cand-rate-not-worse → PRE_EXISTING (report-only, no needs-human); SUSPECT + auto_downgrade + cand-rate-worse → stays SUSPECT+needs-human; SUSPECT + auto_downgrade + thin baseline → stays SUSPECT.
  • class WITHOUT auto_downgrade → SUSPECT unchanged (increment 2 regression guard).

Out of scope

  • Applying auto_downgrade to other classes/agents (dev-lead workload-timeout only initially).
  • Any change to REGRESSION or the reliability gate.

Acceptance

  • lib/ pure; decide_suspect_downgrade unit-tested incl. the tiny-n guard.
  • Only auto_downgrade-flagged SUSPECT classes can downgrade, and only when statistically no-worse-than-baseline on sufficient data; everything else stays SUSPECT+needs-human (increment 2 byte-identical for un-flagged classes).
  • bats green under Canary-rollout Tests; shellcheck clean.
  • Closes the Enhancement: canary gate should assess correctness, not just run-reliability #668 increment series (Layers 0–3 + 1c all delivered).

Activity

  1. don-petry commented on Jul 12, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead Implementation Plan

    Issue: #688 — canary #668 increment 6: SUSPECT→PRE_EXISTING auto-downgrade (decide_suspect_downgrade) — closes #668

    Scope

    Add a default-off, opt-in-per-suspect-class data-driven SUSPECT→PRE_EXISTING auto-downgrade: when a candidate's suspect-class failure rate is statistically no-worse-than the prior version's over the baseline window (and the baseline isn't too thin), the mechanical "environmental, not a regression" judgment is auto-made; genuinely ambiguous/worse cases still stop for a human.

    Implementation Checklist

    • scripts/lib/canary-rollout.sh — add pure decide_suspect_downgrade <cand_permille> <base_permille> <base_sample> <knobs_json> → DOWNGRADE|HOLD (tiny-n guard first, then cand ≤ base+margin, else HOLD)
    • scripts/canary-rollout.sh — add _suspect_downgrade_patterns (auto_downgrade-flagged classes only) + _suspect_class_counts (matched-failures/executed over a window)
    • scripts/canary-rollout.sh — in _frontier_state, when triage==SUSPECT (failure-class, not correctness-shift) and the matched class opts in, compare rates and override SUSPECT→PRE_EXISTING on DOWNGRADE; emit a ::notice::; plumb rate fields on the state line
    • scripts/canary-rollout.sh — update the 3 _frontier_state read sites + _blocker_body (render rate comparison + auto-clear note for a downgraded PRE_EXISTING; no needs-human)
    • standards/canary-rings.json — add auto_downgrade {min_baseline_sample:10, margin_permille:100} to dev-lead's workload-timeout class only

    Tests to Write (TDD, before implementation)

    • decide_suspect_downgrade: DOWNGRADE (cand ≤ base+margin, sample≥min); HOLD (cand worse); HOLD (tiny-n baseline); boundary exactly at base+margin
    • orchestrator: SUSPECT + auto_downgrade + no-worse rate → PRE_EXISTING (report-only, no needs-human); worse rate → stays SUSPECT+needs-human; thin baseline → stays SUSPECT
    • regression guard: class WITHOUT auto_downgrade → SUSPECT unchanged (increment 2 byte-identical)
    • registry: dev-lead workload-timeout carries auto_downgrade; it is the only opted-in class
    • sync-issues: a downgraded SUSPECT files a PRE_EXISTING blocker (no needs-human) with the rate comparison

    Test Command

    bats tests/canary_rollout.bats (plus bash .dev-lead/scripts/dev-lead-lint.sh and shellcheck)

  2. added a commit that references this issue on Jul 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    dev-leadFor dev-lead agent pickup

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions