[Phase 1] Build the deterministic eval scorer + offline tests (triage)

Story

As a skill maintainer,
I want a scorer that runs a skill against its held-out cases and emits a reproducible pass/score report, with offline tests,
so that any candidate version of a skill can be scored deterministically against the fixed held-out set.

Acceptance Criteria

skills/evals/run-eval.sh loads that skill's cases.jsonl, invokes the skill prompt per case, compares output to the expected fields, and prints a per-case result plus an aggregate score as JSON.
For triage, scoring is deterministic: parse the model's JSON output and compare escalate (and risk) to expected; a mismatch or unparseable output counts as a failure.
The scorer invokes the skill through the existing engine abstraction (scripts/engine.sh run_triage) rather than reimplementing model invocation, and accepts an env override for the engine command so tests run without a real provider.
Offline bats tests drive run-eval.sh against a stubbed engine + fixture cases with no network (mirroring the DRY_RUN-offline pattern of tests/test_initiative_planner.bats) and are added to the lint.yml bats list.
run-eval.sh passes shellcheck --severity=warning -x; any python helper follows the repo's inline-pip convention.

Tasks / Subtasks

Write skills/evals/run-eval.sh that sources scripts/engine.sh and iterates a skill's cases (AC: test issue from agent #1, Add @claude delegation, auto-merge, and rebase handling #3)
- Pipe each case's PR context into run_triage with prompts/triage.md
- Allow REVIEW_ENGINE / engine-command override for offline runs
Implement the deterministic triage comparator (escalate + risk equality) (AC: test issue from agent #1, Go-live improvements for PR review agent #2)
Add a stub engine + fixtures and tests/test_skill_evals.bats (AC: Optimize review: small-PR and incremental fast paths #4)
- Assert correct outputs score pass
- Assert a wrong output scores fail
Register the new bats file in .github/workflows/lint.yml and confirm shellcheck-clean (AC: Optimize review: small-PR and incremental fast paths #4, feat: add Copilot engine support via REVIEW_ENGINE toggle #5)

Dev Notes

scripts/engine.sh already exposes run_triage (no-tool tier, stdout capture) and honors REVIEW_ENGINE/per-tier overrides — source it, do not reinvent model invocation. Triage runs on a Haiku-tier model per set_engine_config.
Make the engine command injectable (env override) exactly as engine.sh allows REVIEW_ENGINE to be overridden, so the bats tests run fully offline with a stub — copy the offline discipline of tests/test_initiative_planner.bats (no network in CI).
Emit machine-readable output (aggregate score JSON); reuse the ::error::-style failure messaging convention from validate-plan.py where a hard failure is warranted.
Keep cases read-only here: the scorer reads skills/evals//cases.jsonl but never writes it. Optional cost capture via TOKEN_LOG_FILE (scripts/lib/token-metrics.sh) — leave unset for zero overhead.
Follow AGENTS.md bash standards: #!/usr/bin/env bash, set -euo pipefail, shellcheck source= annotations for engine.sh.

Project Structure Notes

New scorer lives beside its data under skills/evals/; tests added under tests/ alongside existing bats suites and registered in lint.yml exactly like test_initiative_planner.bats.

References

scripts/engine.sh#run_triage
scripts/engine.sh#set_engine_config
scripts/initiative-planner/validate-plan.py
tests/test_initiative_planner.bats
.github/workflows/lint.yml#bats
scripts/lib/token-metrics.sh
AGENTS.md

Likely target surface

skills/evals/run-eval.sh
tests/test_skill_evals.bats
.github/workflows/lint.yml

Story prepared by the BMAD Scrum Master (Bob) for epic #581. Status: ready-for-dev.

Dev-Lead Implementation Plan

Issue: #583 — [Phase 1] Build the deterministic eval scorer + offline tests (triage)

Scope

Build a deterministic eval scorer that runs the triage skill against its held-out cases.jsonl via the existing scripts/engine.sh run_triage abstraction and emits a reproducible per-case + aggregate score as JSON, plus fully-offline bats tests driven by a stub engine.

Placement decision (deviation from the literal issue text)

The issue names skills/evals/run-eval.sh, but epic #581 decision A1 (landed in the #582 commit) relocated the held-out cases to repo-root evals/, and .github/CODEOWNERS + evals/README.md require the scorer to stay separate from the CODEOWNER-gated held-out set (reward-hacking guard). Putting a scorer inside /evals/ would break that separation. The scorer therefore lives at scripts/evals/run-eval.sh — under scripts/ (org "shell orchestration" home), automatically covered by the scripts/**/*.sh shellcheck gate, and reading evals/<skill>/cases.jsonl read-only.

Implementation Checklist

scripts/evals/run-eval.sh — run-eval.sh <skill> (default triage); sources scripts/engine.sh (init output to stderr), iterates evals/<skill>/cases.jsonl, builds prompt = prompts/<skill>.md + ## Pre-fetched PR context + case input, invokes the triage engine command, parses model JSON, compares escalate + risk, emits aggregate JSON with a cases[] array.
- Engine-command env override (EVAL_ENGINE_CMD, default run_triage) so tests run offline with a stub — mirrors how engine.sh honors REVIEW_ENGINE.
- EVALS_DIR override (default repo-root evals) so tests point at fixture cases; cases are read-only.
- Deterministic comparator: parse model JSON; mismatch OR unparseable/non-JSON OR engine non-zero = case failure.
- ::error::-style hard-failure messaging (bad usage / missing prompt / missing cases) per validate-plan.py convention; tiered exit (0 all-pass, 1 some-fail, 2 hard-error).
.github/workflows/lint.yml — add tests/test_skill_evals.bats to the bats list.

Tests to Write (`tests/test_skill_evals.bats`, offline, no network)

Correct stub output scores pass (aggregate score==1, cases[].pass==true).
Wrong stub output (escalate/risk mismatch) scores fail.
Unparseable / non-JSON stub output scores fail.
Mixed fixture set yields the right aggregate counts/score.
Hard error (missing skill arg / unknown skill) exits non-zero with ::error::.

Test Command

bats tests/test_skill_evals.bats (full suite: the bats list in .github/workflows/lint.yml); lint: bash .dev-lead/scripts/dev-lead-lint.sh

Dev-Lead: Implementation Complete

Plan Execution

scripts/evals/run-eval.sh — sources scripts/engine.sh, iterates evals/<skill>/cases.jsonl (read-only), inlines each case input under ## Pre-fetched PR context, invokes the engine command, and emits a single aggregate JSON object with a per-case cases[] array.
Deterministic triage comparator — parses the model's JSON and compares escalate and risk; a mismatch OR unparseable/non-JSON output OR a non-zero engine exit is a case failure.
Engine abstraction via run_triage (AC Add @claude delegation, auto-merge, and rebase handling #3) with an EVAL_ENGINE_CMD env override (default run_triage) so tests run offline with a stub — no model invocation reimplemented.
tests/test_skill_evals.bats — stub engine + fixture cases, fully offline (mirrors the DRY_RUN discipline of test_initiative_planner.bats).
Registered tests/test_skill_evals.bats in .github/workflows/lint.yml bats list.
shellcheck --severity=warning -x clean; bash .dev-lead/scripts/dev-lead-lint.sh passes.

Placement note (deviation from the literal issue text)

The issue named skills/evals/run-eval.sh, but epic #581 decision A1 (landed in the #582 commit) relocated the held-out cases to repo-root evals/, and .github/CODEOWNERS + evals/README.md require the scorer to stay separate from the CODEOWNER-gated held-out set (reward-hacking guard). Placing a scorer inside /evals/ would break that separation, so the scorer lives at scripts/evals/run-eval.sh — under scripts/ (org shell-orchestration home, auto-covered by the scripts/**/*.sh shellcheck gate) and reads evals/<skill>/cases.jsonl read-only.

Test Results

# bats tests/test_skill_evals.bats
1..9
ok 1 scores a correct stub as all-pass (score 1, exit 0)
ok 2 per-case results carry id, pass, expected and actual decisions
ok 3 a wrong escalate/risk decision scores fail (exit 1)
ok 4 unparseable (non-JSON) model output counts as a failure
ok 5 aggregate score is passed/total across a mixed set
ok 6 scorer never writes to the held-out cases file
ok 7 missing skill argument is a hard error (::error::, non-zero)
ok 8 unknown skill (no cases.jsonl) is a hard error (::error::, non-zero)
ok 9 default engine command is run_triage from scripts/engine.sh

# Full CI bats list (all suites): 1..295 — 0 failures
# shellcheck --severity=warning -x scripts/evals/run-eval.sh -> CLEAN
# bash .dev-lead/scripts/dev-lead-lint.sh -> all checks passed

Files Changed

scripts/evals/run-eval.sh (new): deterministic eval scorer.
tests/test_skill_evals.bats (new): 9 offline tests (stub engine + fixtures).
.github/workflows/lint.yml: added tests/test_skill_evals.bats to the bats list.

Notes

Exit codes: 0 all cases pass, 1 ≥1 case failed (gateable), 2 hard error (bad usage / missing prompt or cases file / missing jq). Hard failures use the ::error:: convention from validate-plan.py.
Smoke-checked against the real 8 seed cases via a fixed stub (offline): total=8, scorer parses and scores every line without writing the set.
The bats file declares bats_require_minimum_version 1.5.0 for run --separate-stderr (keeps the scorer's stderr diagnostics out of the asserted JSON). CI's apt bats on ubuntu-latest is 1.10.0.
Optional TOKEN_LOG_FILE cost capture is honored transparently by engine.sh; left unset here for zero overhead.
Only triage exists today; the scorer is keyed on <skill> so additional skills slot in once their evals/<skill>/cases.jsonl + prompts/<skill>.md land.

feat: implement issue #583 — [Phase 1] Build the deterministic eval s…

feat: implement issue #583 — deterministic eval scorer (triage) over #…

fix(evals): tolerate fenced/wrapped JSON in the deterministic scorer …

feat: implement issue #583 — deterministic eval scorer (triage) over #…

Navigation Menu

Repository navigation

[Phase 1] Build the deterministic eval scorer + offline tests (triage) #583

Story

Acceptance Criteria

Tasks / Subtasks

Dev Notes

Project Structure Notes

References

Likely target surface

Dev-Lead Implementation Plan

Scope

Placement decision (deviation from the literal issue text)

Implementation Checklist

Tests to Write (`tests/test_skill_evals.bats`, offline, no network)

Test Command

Dev-Lead: Implementation Complete

Plan Execution

Placement note (deviation from the literal issue text)

Test Results

Files Changed

Notes

13 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Phase 1] Build the deterministic eval scorer + offline tests (triage) #583

Description

Story

Acceptance Criteria

Tasks / Subtasks

Dev Notes

Project Structure Notes

References

Likely target surface

Activity

don-petry commented on Jun 14, 2026

Dev-Lead Implementation Plan

Scope

Placement decision (deviation from the literal issue text)

Implementation Checklist

Tests to Write (tests/test_skill_evals.bats, offline, no network)

Test Command

don-petry commented on Jun 14, 2026

Dev-Lead: Implementation Complete

Plan Execution

Placement note (deviation from the literal issue text)

Test Results

Files Changed

Notes

13 remaining items

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Tests to Write (`tests/test_skill_evals.bats`, offline, no network)