Skip to content

canary-rollout can't promote the #482 reusables: agent dispatch input hardcoded to [dev-lead] + no scheduled promote path #1045

Closed
Closed
canary-rollout can't promote the #482 reusables: agent dispatch input hardcoded to [dev-lead] + no scheduled promote path#1045
Labels
dev-leadFor dev-lead agent pickupenhancementFeature requests

Description

@don-petry
don-petry
opened on Jul 3, 2026

#1044 populated the registry (7 agents now). But the automation still can't COMPLETE the rollout — it can only evaluate, and only promote dev-lead:

🔴 The agent workflow_dispatch input is a hardcoded choice list of [dev-lead]

gh workflow run canary-rollout.yml -f command=evaluate -f agent=auto-rebase → HTTP 422: 'auto-rebase' not in the list of allowed values. So manual evaluate/promote/rollback can only ever target dev-lead. The 6 #482 reusables (4 at ring1 waiting to reach stable) cannot be promoted at all.
Fix: the agent input options: must list every agent in canary-rings.json (or become a free-text input validated against the registry). Ideally generate it from the registry so it never goes stale again.

🟡 No path for the timer to promote (decide the model)

The 4h schedule is read-only evaluate-all (never promotes) by design. So promotion is ONLY manual workflow_dispatch command=promote agent=<x>. Per the owner directive earlier ("complete rollout unless there's a failure to meet criteria" — autonomous through stable, override-only), consider a gated scheduled promote: on the timer, for each agent whose gate PROMOTEs, advance one ring (respecting control holds/overrides), still hard-stopping on regression. Otherwise every ring hop for every reusable is a manual dispatch forever. If manual-only is intended, that's fine — but then "flip to auto" means a human dispatches promote per agent per ring, which should be documented.

Impact

Until at least the agent input lists all 7, the in-flight rollout (auto-rebase, dependabot-automerge, dependabot-rebase, pr-review-mention @ ring1) cannot be carried to stable by the automation. This is now the last blocker to finishing the #482 rollout.

Refs #1036/#1044 (registry), #1025/#1026, #548.

Activity

  1. don-petry
    added
    enhancementFeature requests
    dev-leadFor dev-lead agent pickup
    on Jul 3, 2026

  2. don-petry commented on Jul 3, 2026

    @don-petry
    don-petry
    on Jul 3, 2026
    CollaboratorAuthor

    Dev-Lead Implementation Plan

    Issue: #1045 — canary-rollout can't promote the #482 reusables: agent dispatch input hardcoded to [dev-lead] + no scheduled promote path

    Scope

    Make the canary-rollout.yml agent dispatch input list every agent in canary-rings.json (registry-driven, drift-guarded so it never goes stale), and add a gated scheduled promote path so the 4h timer autonomously advances each agent one ring when its #548 gate PROMOTEs — hard-stopping on regression and respecting control holds/overrides (the owner directive: autonomous through stable, override-only).

    Implementation Checklist

    • scripts/canary_agent_options.sh — new: list (emit sorted registry agent keys = canonical choice options) + check <workflow> (fail on drift vs the workflow's agent input options)
    • scripts/canary-rollout.sh — add cmd_promote_all (fleet-wide gated promote: PROMOTE-only, respects control.hold/control.allow_pre_existing, hard-stops on REGRESSION, honors --dry-run); add _require_agent registry validation to evaluate/promote/rollback; append each real promotion to CANARY_PROMOTIONS_LOG for multi-deploy recording; update usage header
    • .github/workflows/canary-rollout.yml — regenerate agent input options: to all 7 agents; switch the schedule from read-only evaluate-all to gated promote-all (real, not dry); loop the deployment-recording step over CANARY_PROMOTIONS_LOG; update header comments
    • AGENTS.md — document the manual "flip to auto" path + the control.hold pause switch

    Tests to Write

    • tests/canary_rollout.bats: workflow agent options == registry keys (drift guard)
    • tests/canary_rollout.bats: canary_agent_options.sh list emits all 7 sorted keys; check passes on the real workflow and fails on an injected drift
    • tests/canary_rollout.bats: promote-all iterates the registry and advances a PROMOTE-ready agent (dry-run), skips a control.hold agent, and refuses a BLOCKED+REGRESSION agent
    • tests/canary_rollout.bats: unknown agent is rejected with a clear error (registry validation)

    Test Command

    scripts/run-bats.sh tests/canary_rollout.bats and bash .dev-lead/scripts/dev-lead-lint.sh

  3. don-petry commented on Jul 3, 2026

    @don-petry
    don-petry
    on Jul 3, 2026
    CollaboratorAuthor

    Dev-Lead: Implementation Complete

    Plan Execution

    • 🔴 agent dispatch input now lists all 7 registry agents (was hardcoded [dev-lead]) — dispatching -f agent=auto-rebase etc. no longer 422s
    • Registry-derived + drift-guarded so it never goes stale: scripts/canary_agent_options.sh (list regenerates, check fails on drift vs canary-rings.json)
    • Runtime _require_agent validation in canary-rollout.sh (evaluate/promote/rollback reject an unknown agent instead of silently resolving to an empty frontier)
    • 🟡 Model decided: gated scheduled promote (per the "autonomous through stable, override-only" owner directive). The 4h timer now runs promote-all: for every agent, advance one ring per tick iff its feat: implement issue #545 — Compliance: non-stub-dependency-audit.yml #548 gate is PROMOTE
    • promote-all skips gate.control.hold agents, honours gate.control.allow_pre_existing, hard-stops on REGRESSION, never auto---overrides
    • Each real promotion journaled to CANARY_PROMOTIONS_LOG; deployment-recording step loops it so a fleet tick that advances several agents records a Deployment for each
    • Documented the "flip to auto" model + the control.hold pause switch in AGENTS.md

    Test Results

    62 tests, 0 failures  (tests/canary_rollout.bats — 11 new)
  52 evaluate rejects an agent not in the registry
  53 promote rejects an agent not in the registry
  54 canary_agent_options: list emits every registry agent, sorted
  55 canary_agent_options: default (no args) behaves like list
  56 canary_agent_options: check PASSES against the live canary-rollout.yml (no drift)
  57 canary_agent_options: check FAILS when the workflow options drift from the registry
  58 canary-rollout.yml: agent dispatch options == registry keys (drift guard)
  59 promote-all advances a PROMOTE-ready agent (dry-run), iterating the registry
  60 promote-all skips an agent held by control.hold
  61 promote-all does NOT advance a BLOCKED+REGRESSION agent
  62 promote-all records each real promotion to CANARY_PROMOTIONS_LOG

dev-lead-lint.sh: all checks passed (shellcheck 86 files @ warning + agent frontmatter)
YAML parse OK · deployment-step heredoc bash -n OK + functional (valid JSON body per journal line)

    Files Changed

    • .github/workflows/canary-rollout.yml: agent options → all 7 (sorted); schedule evaluate-all → gated promote-all (real, not dry); CANARY_PROMOTIONS_LOG env; deployment step loops the journal (one Deployment per promoted agent); header/comment updates
    • scripts/canary-rollout.sh: _require_agent, cmd_promote_all, promote-all subcommand, promotion journaling (replaces the single-triple GITHUB_OUTPUT), usage header
    • scripts/canary_agent_options.sh (new): list / check — the registry-driven option generator + drift gate
    • tests/canary_rollout.bats: 11 new tests
    • AGENTS.md: new "Canary rollout automation" subsection

    Notes

    • The in-flight feat: implement issue #398 — Compliance: secret_scanning_ai_detection #482 rollout (auto-rebase, dependabot-automerge, dependabot-rebase, pr-review-mention @ ring1) can now be carried to stable — both by manual -f agent=<x> dispatch and autonomously by the timer.
    • The choice-input options: list must stay a single-line flow sequence ([a, b, ...]); the drift checker enforces sync but expects that format. Regenerate with scripts/canary_agent_options.sh list.
    • Concurrency: scheduled promote-all groups under canary-rollout-fleet while manual per-agent promotes group under canary-rollout-<agent>, so a manual promote of the same agent could overlap a fleet tick. Pre-existing group design (fleet was already read-only evaluate-all); tag moves are -f idempotent so the window is low-risk, but a follow-up could tighten it if desired.
    • gate.control.hold is read with a // false default and is not added to the shipped registry — an operator sets it per-agent to pause scheduled promotion without a workflow edit.

  4. don-petry commented on Jul 3, 2026

    @don-petry
    don-petry
    on Jul 3, 2026
    CollaboratorAuthor

    Owner decision: A — gated scheduled auto-promote (truly hands-off, override-only)

    Build both parts:

    1. agent dispatch input lists all registry agents

    Generate the options: from canary-rings.json (or free-text validated against it) so manual evaluate/promote/rollback can target any of the 7 and it never goes stale.

    2. Gated scheduled auto-promote (the timer advances the fleet)

    On the schedule, for each agent in the registry:

    • Evaluate the feat: implement issue #545 — Compliance: non-stub-dependency-audit.yml #548 gate at its current frontier.
    • If gate = PROMOTE (dwell + sample + cumulative-health all pass, benign-allowlist applied) AND no control hold/pin/pause for that agent → advance ONE ring (cut-release.sh <agent> <ver> --channel <next> --promote --push via the App token) and record a GitHub Deployment.
    • Advance at most one ring per agent per run — let each tier soak its own dwell before the next hop.
    • Hard-stop on REGRESSION: never promote; auto-write a control hold + alert (open/label an issue). This is the ONLY thing that stops the rollout — a genuine criteria failure.
    • PRE_EXISTING/benign failures → the benign allowlist already excludes configured classes; anything left un-excluded holds that agent (report, don't promote) and surfaces for a human — do not auto-rollback.

    Safety / control

    • Honor control.yml (or the equivalent): global pause, per-agent hold_at, pin, rollback, force. Default = fully autonomous through stable (override-only).
    • Keep an explicit arm switch (e.g. auto_promote: true default, or a repo variable) so a human can flip the whole loop back to evaluate-only in one place.
    • Still emit the evaluate report every run (observability) whether or not it promotes.

    This is the "complete the rollout unless there's a failure to meet criteria" model the owner asked for. It completes the in-flight #482 rollout (4 reusables at ring1 → stable, one ring per timer as each soaks) without any manual dispatch, while a regression or a control hold stops it cleanly.

    Refs #1036/#1044 (registry), #1025/#1026, #548, #502 (Deployments/observability).

  5. don-petry
    added 4 commits that reference this issue on Jul 3, 2026

    ci(canary-rollout): list all registry agents in the dispatch input (#…

    3a971b6

    feat: implement issue #1045 — canary-rollout can't promote the #482 r…

    049b918

    feat(canary-rollout): gated scheduled fleet auto-promote (#1045 part …

    87adebf

    feat(canary-rollout): gated scheduled fleet auto-promote (#1045 part …

    851f25d
  6. don-petry
    added 6 commits that reference this issue on Aug 2, 2026

    ci(canary-rollout): list all registry agents in the dispatch input (#…

    e8c6678

    feat(canary-rollout): gated scheduled fleet auto-promote (#1045 part …

    9c12d20

    ci(canary-rollout): list all registry agents in the dispatch input (#…

    bffece6

    feat(canary-rollout): gated scheduled fleet auto-promote (#1045 part …

    a17975d

    ci(canary-rollout): list all registry agents in the dispatch input (#…

    1fff0af

    feat(canary-rollout): gated scheduled fleet auto-promote (#1045 part …

    3f70371

  7. 19 remaining items

  8. don-petry
    added 11 commits that reference this issue on Aug 7, 2026

    feat(canary-rollout): gated scheduled fleet auto-promote (#1045 part …

    fd3fdc0

    ci(canary-rollout): list all registry agents in the dispatch input (#…

    cf15db3

    feat(canary-rollout): gated scheduled fleet auto-promote (#1045 part …

    fc84e66

    ci(canary-rollout): list all registry agents in the dispatch input (#…

    7adfd25

    feat(canary-rollout): gated scheduled fleet auto-promote (#1045 part …

    df46f73

    ci(canary-rollout): list all registry agents in the dispatch input (#…

    42615a7

    feat(canary-rollout): gated scheduled fleet auto-promote (#1045 part …

    2205dd3

    ci(canary-rollout): list all registry agents in the dispatch input (#…

    9afb166

    feat(canary-rollout): gated scheduled fleet auto-promote (#1045 part …

    6beef77

    ci(canary-rollout): list all registry agents in the dispatch input (#…

    0aa4539

    feat(canary-rollout): gated scheduled fleet auto-promote (#1045 part …

    653d795
  9. don-petry
    added 2 commits that reference this issue on Aug 18, 2026

    ci(canary-rollout): list all registry agents in the dispatch input (#…

    c7928f6

    feat(canary-rollout): gated scheduled fleet auto-promote (#1045 part …

    1ce5e7c

  10. don-petry commented on Oct 21, 2026

    @don-petry
    don-petry
    on Oct 21, 2026
    CollaboratorAuthor

    Closing after the 2026-10-21 backlog triage.

    Why: Fixed: agent choice lists all registry agents; scheduled promote-all exists.
    Evidence: combsmith/.github canary-rollout.yml:72-84; PRs #1047,#1056,#1060

    Reopen if this still reproduces on main.

  11. don-petry
    closed this as completedon Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    dev-leadFor dev-lead agent pickupenhancementFeature requests

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions