Skip to content

canary-rollout post-merge: fleet-wide evaluate coverage (P1) + benign-failure allowlist + app-id→client-id #1025

Closed
#1026
Closed
canary-rollout post-merge: fleet-wide evaluate coverage (P1) + benign-failure allowlist + app-id→client-id#1025
#1026
Labels
dev-leadFor dev-lead agent pickupenhancementFeature requests

Description

@don-petry
don-petry
opened on Jul 2, 2026

#1002 merged — App-token auth works (token minted, hard-requirement verify passed, run succeeded) and the #548 gate evaluates correctly (validated dev-lead's ring0→ring1 BLOCKED, cum_fail=64, triage=PRE_EXISTING against reality — matches its known benign failure classes). Three follow-ups from the first live report-only run:

🔴 P1 — the scheduled run only evaluates dev-lead, not the fleet

canary-rollout.yml schedule sets AGENT: ${{ github.event.inputs.agent || 'dev-lead' }}, and evaluate requires a single <agent> (usage: evaluate <agent>). So the 4h timer evaluates only dev-lead — the actual fleet (agent-shield, auto-rebase, dependency-audit, dependabot-automerge, dependabot-rebase, pr-review-mention, + onboarding: initiative-planner/idea-triage) is never evaluated or promoted on the timer. This defeats "fleet-wide."
Fix: iterate ALL agents from canary-rings.json on the schedule — either a matrix over the registry keys, or an evaluate-all mode that loops every reusable. (The 4 in-flight #482 reusables at ring1 won't auto-complete until this lands.)

🟡 P2 — PRE_EXISTING failures block advancement indefinitely (needs a benign-class allowlist)

dev-lead's gate is BLOCKED with cum_fail=64 triaged PRE_EXISTING. Per #548, PRE_EXISTING → "do NOT advance, ask human" — correct, but a reusable with recurring benign failures (dev-lead: #864 Dependabot-context dispatch, fix-review git-push-permission — both version-independent) is then permanently blocked with no automated path. Add a per-reusable known-benign failure-class allowlist in canary-rings.json (match by workflow/step/error signature) so those are excluded from cum_fail, and/or a control override to advance-despite-pre-existing. Otherwise every such reusable needs a manual override forever.

⚪ P3 — app-id input deprecated

actions/create-github-app-token warns: Input 'app-id' has been deprecated — use 'client-id'. Switch app-id: ${{ vars.RELEASE_MANAGER_APP_ID }} → client-id: (the App's client ID) when convenient. Non-blocking (still works).

Also: consider setting repositories: on the token step to scope it (currently mints for all petry-projects repos).

Validated live 2026-07-02 against run 28627113475. Refs #999, #1002, #548, #1005.

Activity

  1. don-petry
    added
    enhancementFeature requests
    dev-leadFor dev-lead agent pickup
    on Jul 2, 2026

  2. don-petry commented on Jul 2, 2026

    @don-petry
    don-petry
    on Jul 2, 2026
    CollaboratorAuthor

    Dev-Lead Implementation Plan

    Issue: #1025 — canary-rollout post-merge: fleet-wide evaluate coverage (P1) + benign-failure allowlist + app-id→client-id

    Scope

    Make the 4h scheduled canary-rollout evaluate every agent in canary-rings.json (not just dev-lead), add a per-reusable known-benign failure-class allowlist plus a control.allow_pre_existing advance lever so recurring version-independent failures no longer block forever, and switch the App-token auth from the deprecated app-id to client-id.

    Implementation Checklist

    • P1 scripts/canary-rollout.sh: add evaluate-all subcommand that loops .agents | keys and runs the read-only evaluate for each; update usage/dispatch.
    • P1 .github/workflows/canary-rollout.yml: on schedule, run evaluate-all (fleet-wide) instead of evaluate dev-lead.
    • P2 scripts/lib/canary-rollout.sh: add pure benign_match matcher (workflow + step/error-signature regex).
    • P2 scripts/canary-rollout.sh: exclude allowlisted benign failures from cum_fail — but ONLY when the candidate's reusable is byte-identical to the prior channel (differs=0), so the allowlist can never mask a candidate-introduced regression. Thread a cum_benign count into the evaluate/observability report.
    • P2 standards/canary-rings.json: add gate.benign_failure_classes[] (per reusable) + gate.control.allow_pre_existing.
    • P2 scripts/canary-rollout.sh (cmd_promote): add --allow-pre-existing flag honoring gate.control.allow_pre_existing — advances a BLOCKED+PRE_EXISTING frontier (never REGRESSION).
    • P3 .github/workflows/canary-rollout.yml: switch app-id: ${{ vars.RELEASE_MANAGER_APP_ID }} → client-id: ${{ vars.RELEASE_MANAGER_APP_CLIENT_ID }}; update header/verify-step text. Keep owner:-wide (the stable-tier * member requires org-wide run-history reads, so a fixed repositories: list would break the gate) and document why.

    Tests to Write (tests/canary_rollout.bats)

    • benign_match: workflow+step match → yes; workflow mismatch → no; empty step regex → no (no match-all).
    • orchestrator evaluate-all: loops every registry agent and exits 0.
    • orchestrator: an allowlisted benign in-window failure (reusable unchanged) is excluded → not BLOCKED; the same failure with a changed reusable stays BLOCKED (allowlist disabled when differs=1).
    • orchestrator promote --allow-pre-existing: advances a BLOCKED+PRE_EXISTING frontier but refuses a BLOCKED+REGRESSION one.
    • canary-rings.json: benign_failure_classes + control shape present.

    Test Command

    bats tests/canary_rollout.bats (full suite via scripts/run-bats.sh …), then bash .dev-lead/scripts/dev-lead-lint.sh

  3. don-petry commented on Jul 2, 2026

    @don-petry
    don-petry
    on Jul 2, 2026
    CollaboratorAuthor

    Dev-Lead: Implementation Complete

    Plan Execution

    • P1 evaluate-all subcommand loops .agents | keys — the 4h schedule now evaluates the whole registry, not just dev-lead.
    • P1 canary-rollout.yml schedule forces evaluate-all; concurrency group defaults to fleet for scheduled runs.
    • P2 pure benign_match matcher (workflow ERE + failed-step signature ERE) in the lib decision core.
    • P2 orchestrator excludes allowlisted benign failures from cum_fail, tracked as benign= in the evaluate/[Phase 2] One-action rollback + per-version observability report #502 report — but ONLY when the candidate's reusable is byte-identical to the prior channel (differs=0), so the allowlist can never mask a candidate-introduced regression.
    • P2 benign_failure_classes[] + control.allow_pre_existing added to standards/canary-rings.json (dev-lead's two known-benign classes seeded).
    • P2 promote --allow-pre-existing (honors gate.control.allow_pre_existing) advances a BLOCKED+PRE_EXISTING frontier — never a REGRESSION (that still requires --override).
    • P3 app-id → client-id: ${{ vars.RELEASE_MANAGER_APP_CLIENT_ID }}; header + verify-step text updated. Kept owner:-wide with a documented rationale.

    Test Results

    $ bats tests/canary_rollout.bats
1..51
... (39 pre-existing + 12 new)
ok 40 benign_match: workflow + step signature both match → yes
ok 41 benign_match: workflow regex mismatch → no
ok 42 benign_match: signature does not match step regex → no
ok 43 benign_match: empty step regex never matches (guards against a match-all entry)
ok 44 benign_match: empty workflow regex matches any workflow
ok 45 canary-rings.json: dev-lead gate carries a benign_failure_classes allowlist + control block
ok 46 orchestrator: evaluate-all iterates every agent in the registry (fleet-wide)
ok 47 orchestrator: allowlisted benign failure (reusable unchanged) is excluded from cum_fail → not BLOCKED
ok 48 orchestrator: benign allowlist is DISABLED when the candidate changed the reusable → BLOCKED+REGRESSION
ok 49 orchestrator: a non-allowlisted failure (reusable unchanged) still BLOCKS as PRE_EXISTING
ok 50 orchestrator: promote --allow-pre-existing advances a BLOCKED+PRE_EXISTING frontier (dry-run)
ok 51 orchestrator: promote --allow-pre-existing REFUSES a BLOCKED+REGRESSION frontier

51 tests, 0 failures

    Also green: bash .dev-lead/scripts/dev-lead-lint.sh (shellcheck on 85 files + agent profiles + CODEOWNERS), plus regression runs of test_cut_release.bats, fleet_report.bats, test_bootstrap_new_repo.bats (JSON gate additions don't touch the rings array the bootstrap reads).

    Files Changed

    • scripts/canary-rollout.sh: evaluate-all subcommand + dispatch; benign-allowlist wiring (_benign_patterns/_run_signature/_failure_benign); _cumulative_health now takes apply_benign and returns a benign count; _frontier_state computes differs up front and threads cum_benign; promote --allow-pre-existing.
    • scripts/lib/canary-rollout.sh: pure benign_match; gate_summary_line shows benign=.
    • standards/canary-rings.json: gate.benign_failure_classes[] (2 dev-lead classes) + gate.control.allow_pre_existing + note.
    • .github/workflows/canary-rollout.yml: schedule → evaluate-all; client-id auth; fleet concurrency group; owner-scope rationale.
    • tests/canary_rollout.bats: +12 tests (matcher, SoT shape, fleet iteration, benign exclusion + regression-safety, control override).

    Notes / follow-ups for maintainers

    • Infra prerequisite (P3): a new repo variable RELEASE_MANAGER_APP_CLIENT_ID must be set to the App's client ID (distinct from the old numeric RELEASE_MANAGER_APP_ID). Until it's set, token minting fails hard by design (#868) — set it before/with merge.
    • Benign step regexes are best-effort. I could not fetch the live failed-step names from run 28627113475, so dependabot-context-dispatch/fix-review-git-push-permission match on [Dd]ependabot / [Pp]ush. Please validate against the real failed-step signatures and tune. The differs=0 guard makes a slightly-broad regex safe (it can't mask a candidate regression), and --allow-pre-existing / control.allow_pre_existing is the deterministic fallback lever if a class needs unblocking before its regex is confirmed.
    • Fleet coverage today: canary-rings.json still only registers dev-lead, so evaluate-all currently evaluates one agent — but it now iterates the registry, so the other reusables (agent-shield, auto-rebase, etc.) are picked up automatically once registered, with no further workflow change.
    • Phase 2 (TDD) followed: 12 tests written first and confirmed red before implementing.
  4. don-petry
    closed this as completedin #1026on Jul 3, 2026
  5. don-petry
    added 2 commits that reference this issue on Jul 3, 2026

    feat: implement issue #1025 — canary-rollout post-merge: fleet-wide e…

    7eddaaa

    feat(canary-rings): onboard the 6 #482 reusables into the registry (#…

    4485a5b
  6. don-petry
    added 8 commits that reference this issue on Aug 2, 2026

    feat: implement issue #1025 — canary-rollout post-merge: fleet-wide e…

    383733e

    feat(canary-rings): onboard the 6 #482 reusables into the registry (#…

    b74d8e7

    feat: implement issue #1025 — canary-rollout post-merge: fleet-wide e…

    3180e63

    feat(canary-rings): onboard the 6 #482 reusables into the registry (#…

    9a5ac4a

    feat: implement issue #1025 — canary-rollout post-merge: fleet-wide e…

    c12f6e7

    feat(canary-rings): onboard the 6 #482 reusables into the registry (#…

    f209416

    feat: implement issue #1025 — canary-rollout post-merge: fleet-wide e…

    c007c6a

    feat(canary-rings): onboard the 6 #482 reusables into the registry (#…

    aac63b4

  7. 15 remaining items

  8. don-petry
    added 13 commits that reference this issue on Aug 7, 2026

    feat(canary-rings): onboard the 6 #482 reusables into the registry (#…

    e67290c

    feat: implement issue #1025 — canary-rollout post-merge: fleet-wide e…

    dc7747a

    feat(canary-rings): onboard the 6 #482 reusables into the registry (#…

    028dafc

    feat: implement issue #1025 — canary-rollout post-merge: fleet-wide e…

    979b301

    feat(canary-rings): onboard the 6 #482 reusables into the registry (#…

    a08b663

    feat: implement issue #1025 — canary-rollout post-merge: fleet-wide e…

    5a836e4

    feat(canary-rings): onboard the 6 #482 reusables into the registry (#…

    487c09a

    feat: implement issue #1025 — canary-rollout post-merge: fleet-wide e…

    5183922

    feat(canary-rings): onboard the 6 #482 reusables into the registry (#…

    ccb0fa4

    feat: implement issue #1025 — canary-rollout post-merge: fleet-wide e…

    b8cc6ba

    feat(canary-rings): onboard the 6 #482 reusables into the registry (#…

    6892d4b

    feat: implement issue #1025 — canary-rollout post-merge: fleet-wide e…

    340dfc0

    feat(canary-rings): onboard the 6 #482 reusables into the registry (#…

    ad3c6ca
  9. don-petry
    added 2 commits that reference this issue on Aug 18, 2026

    feat: implement issue #1025 — canary-rollout post-merge: fleet-wide e…

    c831a8a

    feat(canary-rings): onboard the 6 #482 reusables into the registry (#…

    e499986
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    dev-leadFor dev-lead agent pickupenhancementFeature requests

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions