[Phase 2] Extend the harness to prompts/deep-review.md via an LLM-judge scorer #585

Activity

1. github-actions

   added

   initiativeEpic / initiative tracking issueEpic / initiative tracking issue

   on Jun 11, 2026
2. github-actions

   added a parent issue on Jun 11, 2026

   • Initiative: Eval-gated, human-reviewed self-improving skills (SkillOpt-style) #581
3. github-actions

   marked this as blocked on Jun 11, 2026

   • [Phase 1] Build the deterministic eval scorer + offline tests (triage) #583
4. don-petry

   added

   dev-leadFor dev-lead agent pickupFor dev-lead agent pickup

   on Jun 14, 2026
5. don-petry commented on Jun 14, 2026

   

   don-petry

   on Jun 14, 2026

   Collaborator

   More actions

   Dev-Lead Implementation Plan

   Issue: #585 — [Phase 2] Extend the harness to prompts/deep-review.md via an LLM-judge scorer

   Scope

   Add a held-out deep-review eval set plus a config-driven LLM-judge scorer mode to the existing Phase-1 harness, so non-deterministic deep-review output is scored by a Haiku-tier judge selected per-skill (no skill-name branching in the comparator core).

   Path mapping (issue surface → actual repo)

   The issue's skills/evals/* paths map to the repo's real tree:

   • scorer: scripts/evals/run-eval.sh
   • cases: evals/deep-review/{dev,holdout}/cases.jsonl (both splits required by validate-cases.py)
   • judge prompt: evals/judge.md
   • tests: tests/test_skill_evals.bats

   Implementation Checklist

   • evals/deep-review/holdout/cases.jsonl + evals/deep-review/dev/cases.jsonl — held-out cases capturing risk classification, approve-vs-escalate decision, and key findings (grounded in deep-review.md + shared.md risk taxonomy). Disjoint ids across splits.
   • evals/judge.md — versioned, CODEOWNER-gated LLM-judge prompt emitting a numeric score.
   • evals/deep-review/scorer.json — per-skill config selecting llm-judge mode (+ judge prompt + pass threshold). Absent config defaults to deterministic, so triage is unchanged.
   • scripts/evals/run-eval.sh — read per-skill scorer.json; dispatch a generic scorer mode (deterministic vs llm-judge); add an LLM-judge adapter that builds the judge prompt and invokes the judge via EVAL_JUDGE_CMD (default run_triage, Haiku-tier). Per-case numeric score.
   • .github/CODEOWNERS — explicit owner-lock for evals/judge.md (reward-hacking guard, like cases).

   Tests to Write

   • Judge-mode all-pass: stubbed skill + stubbed judge returning high scores → score 1, exit 0.
   • Judge-mode fail: stubbed judge returning a sub-threshold score → case fails, exit 1.
   • Per-case numeric score present in judge-mode results.
   • Judge mode reads only holdout/; never writes the cases file.
   • Unparseable judge output → case failure.
   • Existing deterministic (triage) tests stay green (no skill-name branching regression).
   • validate-cases.py accepts the new deep-review split (committed-tree test).

   Test Command

   bats tests/test_skill_evals.bats tests/test_validate_cases.bats && python3 evals/validate-cases.py && shellcheck scripts/evals/run-eval.sh && bash .dev-lead/scripts/dev-lead-lint.sh
6. don-petry commented on Jun 14, 2026

   

   don-petry

   on Jun 14, 2026

   Collaborator

   More actions

   Dev-Lead: Implementation Complete

   Plan Execution

   • Authored evals/deep-review/holdout/cases.jsonl (5 held-out cases) + evals/deep-review/dev/cases.jsonl (2 proposer-visible cases), capturing the approve-vs-escalate decision, risk classification, escalate_to_opus routing, and the key findings deep-review.md must surface — grounded in the shared risk taxonomy (prompts/shared.md) and deep-review.md's Decision section (AC test issue from agent #1).
   • Added a per-skill scorer-mode config (evals/deep-review/scorer.json) and made the comparator core in scripts/evals/run-eval.sh dispatch on $SCORER_MODE (deterministic | llm-judge) read from that config — no skill-name branching. Absent config defaults to deterministic, so triage is untouched (AC Go-live improvements for PR review agent #2).
   • Added the LLM-judge adapter: it assembles a judge prompt (rubric + expected reference + candidate output) and runs it via EVAL_JUDGE_CMD (default run_triage, the Haiku-tier engine), parsing a numeric per-case score and passing the case when it meets pass_threshold (default 0.7) (AC Add @claude delegation, auto-merge, and rebase handling #3).
   • Added the versioned judge prompt evals/judge.md and CODEOWNER-gated it (/evals/judge.md in .github/CODEOWNERS), alongside the existing holdout gating + the [Guardrail] Hard CI check: fail any proposer-authored PR that touches evals/ #692 CI immutability guard that already covers any evals/ path (AC Add @claude delegation, auto-merge, and rebase handling #3, reward-hacking guard).
   • Added offline judge-mode tests with stubbed skill + stubbed judge (no network) (AC Optimize review: small-PR and incremental fast paths #4).

   Path note

   The issue's "likely target surface" used skills/evals/*; the live tree is scripts/evals/run-eval.sh (scorer) and evals/<skill>/{dev,holdout}/cases.jsonl (cases) from Phase 1. The judge prompt lives at evals/judge.md. Implemented against the live paths.

   Test Results

   bats tests/test_skill_evals.bats
   1..16
   ok 1 scores a correct stub as all-pass (score 1, exit 0)
   ...
   ok 11 llm-judge mode: all-pass scores 1 (exit 0) with per-case numeric score
   ok 12 llm-judge mode: a sub-threshold score fails the case (exit 1)
   ok 13 llm-judge mode: unparseable judge output counts as a failure (score 0)
   ok 14 llm-judge mode: the judge prompt carries both the expected ref and the candidate
   ok 15 llm-judge mode: scorer never writes the held-out cases file
   ok 16 absent scorer.json defaults to deterministic mode (triage unchanged)
   
   python3 evals/validate-cases.py
   OK: 3 skill(s), 21 case(s) across dev+holdout, no cross-split id overlap.
   
   bash .dev-lead/scripts/dev-lead-lint.sh
     [lint] shellcheck on 54 file(s)...
     [lint] validating 4 agent profile(s)...
     [lint] all checks passed
   
   Full bats suite (20 files from lint.yml): 346 tests, 0 failures.
   End-to-end smoke (committed deep-review assets, stubbed skill+judge): 5/5 holdout cases scored, threshold gating correct, exit 0.
   

   Files Changed

   • scripts/evals/run-eval.sh: config-driven scorer mode; score_deterministic + score_llm_judge functions; EVAL_JUDGE_CMD injectable engine command; per-case numeric score; aggregate report shape unchanged (score = passed/total).
   • evals/deep-review/holdout/cases.jsonl, evals/deep-review/dev/cases.jsonl: new held-out + dev splits (disjoint ids).
   • evals/deep-review/scorer.json: selects llm-judge mode + judge prompt + pass threshold.
   • evals/judge.md: versioned LLM-judge rubric emitting {"score": <0..1>, "reason": ...}.
   • .github/CODEOWNERS: owner-lock /evals/judge.md.
   • tests/test_skill_evals.bats: 6 new tests covering judge-mode plumbing (stubbed, offline) + the deterministic-default guarantee.

   Notes

   • Scorer mode is selected purely from scorer.json; a third skill needs only its own config + cases — no edits to the comparator core.
   • Judge runs on the triage (Haiku) tier via run_triage to keep cost low, per the Dev Notes.
   • Unparseable judge output and out-of-range scores are handled (score 0 → fail; scores clamped to [0,1]).
   • Follow-up (out of scope here): deep-review.md writes its verdict to $OUTPUT_FILE rather than stdout, while the scorer (like the Phase-1 triage scorer) captures stdout. Real end-to-end deep-review scoring would need a small prompt/harness adaptation to emit the verdict on stdout; the offline judge plumbing (AC Optimize review: small-PR and incremental fast paths #4) is what this story scopes.
7. don-petry

   added a commit that references this issue on Jun 15, 2026

   feat: implement issue #585 — [Phase 2] Extend the harness to prompts/…

   ...

   c67811a
8. don-petry

   closed this as completedin #745on Jun 15, 2026
9. github-actions

   marked this as blocking on Jun 20, 2026

   • [Phase 1] Validate Opus 4.8 >= Opus 4.7 on the deep-review held-out eval suite (regression gate) #836