Skip to content

[Phase 1.5] Shadow triage: run a candidate triage model alongside the incumbent on real PRs, logged but never acted on #2108

Closed

Description

@github-actions

Story

As a maintainer replacing the routing model in the review cascade,
I want the candidate's triage decision on real PRs logged next to the incumbent's,
so that false-approve risk is measured on real traffic, because a 5-case held-out set cannot show a rise in false-approves.

Acceptance Criteria

  1. When a repository variable naming a shadow candidate model is set, scripts/review-one-pr.sh runs the candidate's triage on the same prompt after the incumbent's and writes the candidate's {escalate, risk} decision and model id to the token-usage record. The incumbent's result alone drives the cascade.
  2. If the variable is unset (the default), behaviour and outputs are unchanged. A bats test asserts this.
  3. Shadow runs stop by themselves once a configured end date (at most 14 days after enabling, per the epic cost cap) has passed, even if the variable is still set.
  4. A shadow failure (non-JSON, refusal, error, rate limit) is logged as a shadow outcome and never changes the PR's review result or exit code.
  5. Shadow token records carry the candidate model id, so token_report.sh prices them under the candidate's row.

Tasks / Subtasks

Dev Notes

  • Triage is invoked and validated in scripts/review-one-pr.sh (the cascade failed at tier 1 handling). The shadow call must sit outside that error path so it can never fail the review.
  • Pin the candidate for the shadow call only, by setting CLAUDE_TRIAGE_MODEL_CHAIN=<candidate> for that single run_triage invocation (model-ab.sh uses the same pattern).
  • Workflow env wiring goes in .github/workflows/pr-review.yml. Follow AGENTS.md for repository variables. Do not touch thin caller stubs.
  • This is pipeline Stage 6 (shadow) from discussion Enable Haiku 5.5 #2095. Keep it model-agnostic.

Project Structure Notes

Inert by default. The kill switch is unsetting the variable.

References

  • scripts/review-one-pr.sh
  • scripts/engine.sh
  • .github/workflows/pr-review.yml
  • scripts/token_report.sh
  • AGENTS.md

Likely target surface

  • scripts/review-one-pr.sh
  • .github/workflows/pr-review.yml

Story prepared by the BMAD Scrum Master (Bob) for epic #2101. Status: ready-for-dev.

Metadata

Metadata

Assignees

No one assigned

    Labels

    initiativeEpic / initiative tracking issue

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions