Autonomous fleet-wide soak-and-promote workflow for all first-party reusables

Successor to the closed #501 ("ring-staged, health-gated promotion"): build the autonomous loop that soaks and promotes every first-party reusable through the canary rings, fleet-wide — autonomous by default, human override at any point, with native GitHub traceability. Consumes the graduated gate (combsmith/.github#548) and the cut-release.sh primitives (#959 cross-repo, #992 --promote); delivers #684's org-wide coverage.

Architecture — one scheduled loop (idempotent, stateless)

hourly workflow (GitHub Actions, in .github-private)
  ├─ read control.yml (overrides) + registry.yml (reusables, rings, gate knobs)
  ├─ for each reusable (.github- AND .github-private-hosted):
  │     frontier = highest channel at the candidate
  │     verdict  = soak-check(reusable, source-tier, graduated gate)   # .github#548
  │     if PASS & not held: cut-release.sh <name> <ver> --channel <next-ring> --promote --push
  │     if regression (failure-triage): HALT + auto-hold + alert
  ├─ record each advance as a GitHub Deployment (environment = ring)
  └─ rewrite live status (pinned issue + release/STATUS.md)

Every run re-derives truth from channel tags + run history, so missed/duplicated runs are harmless.

Components

release/registry.yml — all first-party reusables (host repo, ring→repos map, per-reusable gate knobs, soak_start_ring for no-next-caller reusables). Tag-based release pinning for ALL first-party reusable workflows + org-wide compliance enforcement #684 coverage.
soak-check gate engine (.github#548) — (reusable, target_ring) → PASS | WAIT(reason) | HALT(regression): graduated dwell + fresh-sample + cumulative health + robust baseline + failure-triage. Unit-tested; CLI + library.
cut-release.sh --promote (feat(cut-release): add --promote for ring-to-ring channel moves (#501) #992) — channel-move primitive; cross-repo aware (feat: wire cut-release.sh cross-repo publish via gh api (#872) #959); drives both hosts.
Scheduled workflow — runs the loop. Auth via a GitHub App scoped contents:write on .github + .github-private only (least-privilege, auditable, rotatable).

Human override (override-only — never a required gate)

A declarative release/control.yml read every run (editing it = the override, itself version-controlled = traceable):

pause: false                 # global kill switch
reusables:
  pr-review-mention:
    hold_at: ring1           # auto up to ring1, then wait for a human (optional, per-reusable)
    pin: 2.1.1               # never auto-promote past this version
  auto-rebase:
    force: { ring: ring1 }   # one-shot: advance now, bypass gate (loop clears after applying)
    rollback: 2.0.0          # one-shot: move channel back (immutable tag still exists)

Defaults fully autonomous through stable. A detected regression auto-writes a hold_at + opens an alert (fail-safe).

Reporting & traceability (three layers)

GitHub Deployments + Environments — ring = Environment; advance = Deployment whose description embeds the gate evidence (dwell, sample N/target, cumulative-fail-count, authorizing run-window). The Deployments tab = permanent, queryable history (which version reached which ring, when, on what evidence). Environment protection rules are where a required-reviewer could later be bolted on (off per override-only policy).
Live status — pinned tracking issue + committed release/STATUS.md: the promotion matrix (reusable × ring, version, time-in-ring, gate, next-eligible, health) — always current, links to deployments + authorizing runs.
Alerts + per-run job summary — every run writes its evaluation table to the job summary; a HALT opens/updates a labeled issue with the failing run, the regression-vs-pre-existing triage, and the auto-hold it applied, and pings.

Cross-cutting

Dual-host via registry.host + cut-release.sh cross_repo_agent split (.github tags on .github; .github-private's on its origin).
Rollback is free (immutable vX.Y.Z tags) — automatic on regression, or via control.yml.
Canary the canary: ship in report-only mode first (evaluates + writes STATUS, advances nothing), validate against the in-flight Bring #482's .github-hosted reusables under the canary-ring model (extend cut-release.sh; coordinate with #866) #870 promotion for a cycle, then enable auto-advance ring-by-ring (next→ring0, then ring1, then stable).

Build sequence

soak-check engine + registry.yml (the .github#548 standard, graduated gate, robust baseline). ← in progress (feat: implement issue #545 — Compliance: non-stub-dependency-audit.yml #548, dev-lead)
cut-release.sh --promote ← DONE (feat(cut-release): add --promote for ring-to-ring channel moves (#501) #992)
Scheduled workflow + GitHub App + Deployments/STATUS reporting (this issue).
Flip report-only → auto, ring-by-ring.

References

Closed [Phase 2] Replace PUT-contents clobber deploy with versioned, ring-staged, health-gated promotion #501 (predecessor scope). Gate: .github#548. Coverage: Tag-based release pinning for ALL first-party reusable workflows + org-wide compliance enforcement #684. Cross-repo cut: feat: wire cut-release.sh cross-repo publish via gh api (#872) #959. Promote primitive: feat(cut-release): add --promote for ring-to-ring channel moves (#501) #992. In-flight first promotion: Bring #482's .github-hosted reusables under the canary-ring model (extend cut-release.sh; coordinate with #866) #870.

Dev-Lead Implementation Plan

Issue: #993 — Autonomous fleet-wide soak-and-promote workflow for all first-party reusables

Scope

Build the fleet-wide, registry-driven successor to the single-agent canary loop (#501): a registry (release/registry.yml) + declarative override file (release/control.yml) + a pure decision core + an orchestrator that evaluates every first-party reusable each run and rewrites live status — shipped in report-only mode first ("canary the canary"), with the auto-advance path implemented and unit-tested but gated off in the scheduled workflow.

Out of scope (explicit, deferred to build-sequence step 4 / other issues)

GitHub App provisioning (infra prerequisite, not code — workflow consumes a token secret as placeholder).
GitHub Deployments API recording wired live (the record_deployment helper is implemented + tested with stubs but only invoked on a real auto-advance; the scheduled workflow runs report-only so advances/deployments don't fire yet).
The soak-check gate engine itself (.github#548) and cut-release.sh --promote (feat(cut-release): add --promote for ring-to-ring channel moves (#501) #992) live in other issues/repos; this loop reuses the existing decide_gate core + channel-tag-move primitive.

Implementation Checklist

release/registry.yml — fleet registry of first-party reusables (host repo, ordered ring→repos map, per-reusable gate knobs, soak_start_ring).
release/control.yml — declarative override file (global pause; per-reusable hold_at, pin, one-shot force, one-shot rollback) with autonomous-by-default values.
scripts/lib/soak-promote.sh — pure decision core: plan_action state machine (control overrides × gate verdict × ring ceilings), ring_index, version_gt, status_row renderer. Side-effect-free, source-able.
scripts/soak-promote.sh — orchestrator: read registry + control via yq, iterate reusables, derive frontier/candidate/gate, compute action, render release/STATUS.md + job summary; report (default) advances nothing, auto performs the channel-tag move + record_deployment.
release/STATUS.md — committed live promotion matrix (reusable × ring), regenerated each run.
.github/workflows/soak-promote.yml — hourly schedule + workflow_dispatch, runs the orchestrator in report-only mode.
AGENTS.md — document the new repo-specific workflow exception (no org template) + add the new bats file to lint.yml's bats list.

Tests to Write (`tests/test_soak_promote.bats`)

plan_action: PAUSED (global kill switch wins), ROLLBACK/FORCE one-shots, COMPLETE (no frontier), HALT (gate INVESTIGATE), PINNED (candidate > pin), HOLD (next ring past hold_at), WAIT (SOAKING), ADVANCE (PROMOTE + nothing blocks), and precedence ordering.
ring_index / version_gt pure helpers (ordering + semver compare edge cases).
status_row rendering shape.
release/registry.yml + release/control.yml are valid YAML with the expected shape.
Orchestrator report against a fixture registry/control with stubbed git/gh: writes STATUS.md, prints the eval table, and never pushes a tag.
Orchestrator auto on an ADVANCE: invokes the tag move (asserted via stub log); report mode on the same input does not.

Test Command

bats tests/test_soak_promote.bats (full suite via scripts/run-bats.sh), plus bash scripts/dev-lead-lint.sh.

Dev-Lead: Implementation Complete

Built the autonomous fleet-wide soak-and-promote loop as the registry-driven successor to the single-agent canary (#501), shipped in report-only mode ("canary the canary"). The auto-advance path is implemented and unit-tested but the scheduled workflow runs report-only — auto is enabled later, ring-by-ring, via mode=auto dispatch (build-sequence step 4).

Plan Execution

release/registry.yml — fleet registry of all first-party reusables (host, ordered ring→repos map, per-reusable gate knobs, soak_start_ring) — Tag-based release pinning for ALL first-party reusable workflows + org-wide compliance enforcement #684 coverage.
release/control.yml — override-only human control surface (pause, hold_at, pin, one-shot force/rollback), autonomous by default.
scripts/lib/soak-promote.sh — pure decision core (plan_action state machine, ring_index, version_gt, status_row), side-effect-free + source-able.
scripts/soak-promote.sh — orchestrator: re-derives each reusable's frontier from channel tags + run history, decides an action through the overrides, rewrites STATUS; report mutates nothing, auto moves channel tags + records Deployments. Reuses the [Phase 2] Replace PUT-contents clobber deploy with versioned, ring-staged, health-gated promotion #501 decide_gate core.
release/STATUS.md — committed live promotion matrix (regenerated each run).
.github/workflows/soak-promote.yml — hourly + dispatch, runs report-only on the timer.
AGENTS.md exception note + tests/test_soak_promote.bats added to the lint.yml bats list.

Test Results

$ bats tests/test_soak_promote.bats
... 25 tests, 0 failures (plan_action precedence, ring_index, version_gt,
    status_row, registry/control YAML shape, orchestrator report/auto with
    stubbed git/gh, pause kill-switch, empty-candidate-version regression)

$ <full CI bats list via run-bats.sh>
912 tests, 0 failures

$ bash scripts/dev-lead-lint.sh
[lint] all checks passed   (shellcheck --severity=warning -x scripts/**/*.sh; agent frontmatter; CODEOWNERS)

Files Changed

release/registry.yml (new): fleet of 9 reusables across both hosts (this repo + petry-projects/.github).
release/control.yml (new): autonomous-by-default override file.
release/STATUS.md (new): seed promotion matrix.
scripts/lib/soak-promote.sh (new): pure decision core.
scripts/soak-promote.sh (new): the loop orchestrator.
.github/workflows/soak-promote.yml (new): hourly report-only schedule.
tests/test_soak_promote.bats (new): 25 unit + integration tests.
.github/workflows/lint.yml, AGENTS.md: wire the new test + document the workflow exception.

Notes

Report-only first, by design. Per the issue's "canary the canary" directive, the scheduled workflow evaluates + writes STATUS but advances nothing. The auto path (ADVANCE/FORCE/ROLLBACK channel-tag moves + Deployment records) is implemented and covered by tests with stubbed git/gh, but only runs on an explicit mode=auto dispatch.
Deferred (other issues / infra): the GitHub App provisioning (workflow falls back to GH_PAT_WORKFLOWS until then), the soak-check gate engine (.github#548 — this loop reuses the existing decide_gate core), and cut-release.sh --promote (feat(cut-release): add --promote for ring-to-ring channel moves (#501) #992 — the loop moves channel tags directly, as canary-rollout does).
Operational note: the report-only workflow commits the refreshed release/STATUS.md back to the default branch via the checkout's GH_PAT_WORKFLOWS credential; if main branch protection blocks bot pushes, that credential's bypass (or the future App) is required.
Found + fixed during self-review: frontier_state now emits |-delimited fields so an empty candidate version (common cross-repo) cannot shift the positional read (locked in by a regression test).

feat: implement issue #999 — Harden #993 soak-promote to spec: #548 g…

Closing after the 2026-10-21 backlog triage.

Why: Delivered as canary-rollout autocut->promote-all schedule in .github.
Evidence: PR #1002; combsmith/.github canary-rollout.yml cron 33 */4

Reopen if this still reproduces on main.

Navigation Menu

Repository navigation

Autonomous fleet-wide soak-and-promote workflow for all first-party reusables #993

Architecture — one scheduled loop (idempotent, stateless)

Components

Human override (override-only — never a required gate)

Reporting & traceability (three layers)

Cross-cutting

Build sequence

References

Dev-Lead Implementation Plan

Scope

Out of scope (explicit, deferred to build-sequence step 4 / other issues)

Implementation Checklist

Tests to Write (`tests/test_soak_promote.bats`)

Test Command

Dev-Lead: Implementation Complete

Plan Execution

Test Results

Files Changed

Notes

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Autonomous fleet-wide soak-and-promote workflow for all first-party reusables #993

Description

Architecture — one scheduled loop (idempotent, stateless)

Components

Human override (override-only — never a required gate)

Reporting & traceability (three layers)

Cross-cutting

Build sequence

References

Activity

don-petry commented on Jun 30, 2026

Dev-Lead Implementation Plan

Scope

Out of scope (explicit, deferred to build-sequence step 4 / other issues)

Implementation Checklist

Tests to Write (tests/test_soak_promote.bats)

Test Command

don-petry commented on Jun 30, 2026

Dev-Lead: Implementation Complete

Plan Execution

Test Results

Files Changed

Notes

don-petry commented on Oct 21, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Tests to Write (`tests/test_soak_promote.bats`)