Score persona eval sets weekly at parity tier — add the cadence, preserve the daily triage run #1702

Decision

Persona eval sets are scored at parity tier on a WEEKLY cadence (maintainer decision, 2026-09-07).

Correcting the cost basis this decision was first framed against

An earlier estimate described parity as costing ~$158/yr for qa-lead by changing a daily run.
That figure was wrong and is retracted. Verified against main:

• .github/workflows/skill-eval-report.yml runs cron: '51 7 * * *' (daily) but scores
  triage only — SKILL: ${{ inputs.skill || 'triage' }}, and nothing overrides the default on
  the scheduled path.
• No persona is scored on any cron today. Every qa-lead scoring run to date has been a manual
  workflow_dispatch.
• Parity itself is not on main yet: "engine": "persona" lives in unmerged PR feat: implement issue #1686 — Persona eval harness scores the wrong model and the wrong context shape — a scored promotion gate built on it measures neither the shipped persona nor the case #1696 (issue Persona eval harness scores the wrong model and the wrong context shape — a scored promotion gate built on it measures neither the shipped persona nor the case #1686).
  On main, all three scorer.json files (deep-review, qa-lead, solution-architect) declare no
  engine and therefore default to the Haiku-tier run_triage.

So this is net-new recurring spend, not a saving. Corrected figures, on the same assumptions the
eval-cost-estimate.sh script uses (6 held-out cases; 12k/1.2k engine tokens, 3k/300 judge tokens per
case; Opus $5/$25 per Mtok vs Haiku $1/$5):

Weekly was chosen deliberately as a regression tripwire: cheap enough to run unattended, frequent
enough to catch a prompt or vendored-framework change that degrades advice quality between promotion
decisions. The existing daily triage run is unaffected and stays as-is — it is Haiku-tier and cheap.

Acceptance Criteria

1. Persona eval sets are scored weekly, at the tier their scorer.json declares. A skill with
   "engine": "persona" is scored at parity; a skill without it keeps run_triage. No skill is
   silently upgraded — the declaration remains the only thing that selects the tier (Persona eval harness scores the wrong model and the wrong context shape — a scored promotion gate built on it measures neither the shipped persona nor the case #1686 AC Go-live improvements for PR review agent #2).
2. The existing daily triage run is preserved unchanged (cron: '51 7 * * *', Haiku). This
   story adds a persona cadence; it does not repurpose or degrade the existing one.
3. The weekly run discovers which skills to score from their scorer.json (engine == "persona"),
   rather than hardcoding qa-lead. Adding a persona to the parity cadence must require no edit to
   this workflow — the same derive-don't-enumerate rule as the <id>:hands-off label family (fix(pr-review-sweep): dispatch with GH_PAT_WORKFLOWS, not the bot comment PAT #756).
4. Weekly runs are off-peak and do not contend with the existing daily report — pick a distinct
   day+hour and state it, per AGENTS.md "Scheduled workflows".
5. The run is report-only and non-blocking, exactly like the current skill-eval-report path: a
   held-out regression is a ::warning:: and a published report, never a failed required check. It
   must not become a merge gate by accident.
6. The report records which tier each skill was scored at, so a number is never ambiguous about what
   produced it (Persona eval harness scores the wrong model and the wrong context shape — a scored promotion gate built on it measures neither the shipped persona nor the case #1686 AC test issue from agent #1).
7. Cost is re-stated at implementation time using scripts/evals/eval-cost-estimate.sh (added by
   Persona eval harness scores the wrong model and the wrong context shape — a scored promotion gate built on it measures neither the shipped persona nor the case #1686) rather than the estimates above, and the actual per-run figure is recorded in the PR. If the
   real cost diverges materially from ~$22/yr for one persona, say so rather than proceeding silently.

Tasks / Subtasks

• Add the weekly persona-parity schedule, deriving the skill list from scorer.json. (AC: test issue from agent #1, Add @claude delegation, auto-merge, and rebase handling #3, Optimize review: small-PR and incremental fast paths #4)
• Preserve the daily triage path untouched. (AC: Go-live improvements for PR review agent #2)
• Surface the scored tier in the report output. (AC: feat: feature ideation framework — adversarial proposer/challenger/synthesizer #6)
• Keep the path report-only/non-blocking. (AC: feat: add Copilot engine support via REVIEW_ENGINE toggle #5)
• Run eval-cost-estimate.sh and record real figures in the PR. (AC: Workflow should skip no-op PRs in MAX_PRS count #7)

Dev Notes

• Blocked by Persona eval harness scores the wrong model and the wrong context shape — a scored promotion gate built on it measures neither the shipped persona nor the case #1686. Parity (run_persona, the engine field in scorer.json) does not exist on
  main yet — it is in PR feat: implement issue #1686 — Persona eval harness scores the wrong model and the wrong context shape — a scored promotion gate built on it measures neither the shipped persona nor the case #1696. Implementing this first would wire a cadence for a tier that has no
  implementation.
• Do not add a new workflow file if the existing skill-eval-report.yml can carry a second
  schedule entry; AGENTS.md discourages new cron surface, and this initiative has already added
  observability rather than new scheduled workloads elsewhere.
• No new count-based breakers or budget machinery. The existing automation budget and runaway
  detector remain the ceiling; this is a scheduled report, not an agent action loop.
• Note for whoever implements: the current scheduled path scores exactly one skill. Scoring N skills in
  one run needs a loop or a matrix — keep the per-skill report artifact shape (eval-report.json per
  skill) so existing consumers and the skill-eval-report-<skill> artifact naming still work.

References

• .github/workflows/skill-eval-report.yml (cron: '51 7 * * *', SKILL default triage)
• scripts/evals/run-eval.sh, scripts/evals/eval-cost-estimate.sh (Persona eval harness scores the wrong model and the wrong context shape — a scored promotion gate built on it measures neither the shipped persona nor the case #1686)
• evals/<skill>/scorer.json — the engine declaration that selects the tier
• Persona eval harness scores the wrong model and the wrong context shape — a scored promotion gate built on it measures neither the shipped persona nor the case #1686 / PR feat: implement issue #1686 — Persona eval harness scores the wrong model and the wrong context shape — a scored promotion gate built on it measures neither the shipped persona nor the case #1696 (engine parity — the hard dependency), [qa-lead S2] Make the qa-lead eval set scorable and capture a first held-out baseline #1645 / PR feat: implement issue #1645 — [qa-lead S2] Make the qa-lead eval set scorable and capture a first held-out baseline #1656 (per-skill judge)
• Epic: Persona framework — post-rollout hardening & follow-ups #1318 (persona post-rollout tracker — scored eval-gate follow-up)
• AGENTS.md — "Scheduled workflows", "Cost reporting"

Delivery brief — 2026-09-13 (read this section first; implement it, do not re-ask)

"Blocked by #1686" (Dev Notes) is discharged. #1686 is closed, PR #1696 merged 2026-09-08, and scripts/evals/eval-cost-estimate.sh exists on main. Parity is live: on main, every evals/*/scorer.json except deep-review declares "engine": "persona". Discovery (AC #3) therefore yields nine skills: business-analyst, dev-lead, devops-lead, pr-review, qa-lead, scrum-master, security-lead, solution-architect, sre-lead. That matches the "all 9 personas" row of the cost table.

Two hazards the description does not mention

1. Concurrency will cancel runs. skill-eval-report.yml has concurrency: { group: skill-eval-report, cancel-in-progress: true }. As written, the weekly run and the daily triage run cancel each other, and a per-skill matrix in that one group cancels its own siblings. Key the group by event and skill, e.g. skill-eval-report-${{ github.event_name }}-${{ matrix.skill || inputs.skill || 'triage' }}, so AC Go-live improvements for PR review agent #2 (daily run unchanged in behaviour) actually holds.
2. Most of the nine skills' case sets fail case.schema.json today. [qa-lead S8] Fleet-wide: 11 of 12 eval sets fail case.schema.json and CI never notices #1651 owns that, and it is in flight in parallel. One skill erroring must not abort the rest: use fail-fast: false, and record the failure as that skill's row in the report (AC feat: add Copilot engine support via REVIEW_ENGINE toggle #5 report-only). Do not edit anything under evals/*/cases.jsonl or evals/case.schema.json in this PR.

Pre-answered decisions

• Keep one workflow file: add a second schedule entry to skill-eval-report.yml (Dev Notes).
• Weekly slot: 23 9 * * 0 (Sunday 09:23 UTC). It is distinct from the daily 07:51 run and from apply-repo-settings (Monday 06:17). State it in the header comment (AC Optimize review: small-PR and incremental fast paths #4).
• Tell the two schedules apart with github.event.schedule: the daily cron keeps SKILL=triage, and the weekly cron runs the discovered matrix.
• Keep the skill-eval-report-<skill> artifact naming and eval-report.json per skill.
• The timeout-minutes: 20 is per job; with a matrix it applies per skill, which is fine.

Budget guard

1. Schedule, discovery job, matrix, concurrency fix and fail-fast: false (AC test issue from agent #1–feat: add Copilot engine support via REVIEW_ENGINE toggle #5), with a bats or unit test proving that discovery derives the list from scorer.json.
2. Tier recorded in the report output (AC feat: feature ideation framework — adversarial proposer/challenger/synthesizer #6).
3. Run eval-cost-estimate.sh and paste the figure into the PR description (AC Workflow should skip no-op PRs in MAX_PRS count #7). Do not dispatch real parity scoring runs to verify; the estimate script is the required evidence.

If you run low on budget, stop after step 1 or 2 with a green PR and list what remains. A timed-out run that loses the work is worse than a partial.