Skip to content

[Phase 2] Replace PUT-contents clobber deploy with versioned, ring-staged, health-gated promotion #501

Closed

Description

@don-petry

Parent: #495 · Phase 2 · Work type: Infra · Targets SC3, SC5, SC8.

Replace deploy-pr-review.yml/force-deploy-pr-review.yml (PUT /contents clobber of all consumers' main at once) with a versioned, ring-staged promotion workflow whose only action is moving channel tags (git tag -f <ring> <vX.Y.Z> && git push -f). Ring N+1's tag advances only when an automated health gate confirms ring N is healthy (run-success rate, cancellation rate, no new failure-report issues over a soak window). The promotion workflow is the sole authorized mover of channel tags (paired with the tag-protection rules).

Done when: no path overwrites consumer files or >1 ring at a time; promotion is a health-gated tag move.

Ref: §2.3, §5.1, §7 Phase 2.

Activity

  1. github-actions commented on Jun 8, 2026

    @github-actions
    Contributor

    Thanks for opening this and linking it back to the parent initiative — the done-when criteria and the reference to §2.3 give a clear target to build toward. For the health gate, could you share the intended ring topology (e.g., how many rings, which consumers map to each) and the concrete thresholds you have in mind for run-success rate and soak window duration? That'll help scope the gate logic and avoid revisiting those parameters mid-implementation.

  2. don-petry commented on Jun 21, 2026

    @don-petry
    CollaboratorAuthor

    Decisions / clarifications (from design discussion combsmith/.github#516)

    Makes the abstract "health gate over a soak window" concrete.

    Soak / health gate (per ring)

    A ring's channel advances to candidate vX.Y.Z only once the rings already running the candidate have produced:

    • Volume: ≥ the average daily runs over a trailing 7-day moving window of healthy candidate runs — min_healthy_runs = ceil(baseline_runs / 7).
    • Quality: failure rate ≤ baseline (prior version's failure rate over the same window).

    Low-traffic / unused reusables — we still wait, no floor

    If a ring isn't producing healthy candidate runs, the gate simply never advances. No synthetic floor (no "promote after N days regardless", no "floor of 1 run"). Rationale: if a reusable isn't in use, its new version doesn't matter — the candidate just parks at the frontier ring until real volume accrues.

    Failure handling

    • Every soak failure is investigated. If confirmed pre-existing → logged as an issue, does not block promotion.
    • A new regression from the candidate → explicit human override to continue, or (more commonly) fix + new vX.Y.Z which resets the rollout (ring channels roll back to last known-good).

    Evaluator cadence — every 4 hours

    Read-only evaluate runs on a 4h schedule: (+ workflow_dispatch). promote stays dispatch-gated, never on the timer.

    Mover identity — GH_PAT_WORKFLOWS now, GitHub App later

    GH_PAT_WORKFLOWS (org classic PAT, repo scope) is the existing proven cross-repo write credential (used by gh-aw-cross-org) and has the contents:write needed to force-update channel refs. No new secret. Long-term hardening: dedicated GitHub App w/ only contents:write via actions/create-github-app-token (the petry-projects-planner pattern). Caveat: verify token write-membership before any don-petry-org targets.

    Suggested implementation surface

    • scripts/lib/canary-rollout.sh — pure, unit-tested decision core (daily-avg threshold, failure-rate compare, gate state machine: PROMOTE / SOAKING / INVESTIGATE / RESET).
    • scripts/canary-rollout.sh — evaluate (read-only report) + promote (gated tag move, --override, --dry-run).
    • .github/workflows/canary-rollout.yml — 4-hourly evaluate + dispatch-gated promote.
    • test/scripts/canary-rollout/*.bats (TDD with gh stubs).

    Tag-protection coupling (the promotion workflow as sole authorized mover) is tracked in the new ruleset follow-up child issue.

  3. 60 remaining items

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementFeature requestsinitiativeEpic / initiative tracking issue

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions