Skip to content

[Phase D] Post-promotion watch: restore the previous default and pause further rollouts on a regression beyond tolerance #2129

Open

Description

@github-actions

Story

As a maintainer trusting automated promotion,
I want the same metrics monitored after a model is promoted, with automatic rollback on a regression,
so that a wrongly promoted model is reverted within one monitoring interval.

Acceptance Criteria

  1. A scheduled watch evaluates, for each (family, tier) whose head was changed by the pipeline, the canary metrics (cost USD, latency p50/p95, refusal and error rate, spike cap) against the policy tolerances, using the previous default's records as the baseline.
  2. On a regression beyond tolerance, the watch opens and merges a PR that restores the previous default as the chain head, moves the channel back with scripts/cut-release.sh --promote to the prior release, pauses further rollouts for that (family, tier), and records the reason on the onboarding issue with the for-human-analysis label.
  3. The watch's evaluation is a pure function with bats covering no regression, cost regression, latency regression, refusal spike and INSUFFICIENT data (which never triggers a rollback).
  4. The monitoring interval is read from the policy file, and the kill switch stops the watch from acting.

Tasks / Subtasks

Dev Notes

  • Split from auto-promotion at the maintainer's request so each can be reviewed alone.
  • Related existing work: [Phase 2] Test coverage for canary-rollout's one-action rollback (SC4 guard) #1623 (test coverage for canary-rollout's one-action rollback). Reuse that rollback path rather than inventing one.
  • The monitoring-interval value is a policy key; if the policy story did not define one, add it there via this PR's policy change and record it in the ADR.
  • Pure-logic-plus-bats (ADR-0004).
  • Uses the same merge- and tag-push-capable credential as auto-promotion (epic untracked prerequisites).

Project Structure Notes

One watch workflow plus a pure evaluator library.

References

Likely target surface

  • scripts/lib/
  • .github/workflows/

Story prepared by the BMAD Scrum Master (Bob) for epic #2113. Status: ready-for-dev.

Metadata

Metadata

Assignees

No one assigned

    Labels

    initiativeEpic / initiative tracking issue

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions