[Phase 1] Define held-out eval-case format + seed prompts/triage.md case set

Story

As a skill maintainer,
I want a versioned, documented held-out eval-case format plus a seed case set for the triage skill,
so that skill quality can be measured against a fixed, reward-hack-resistant set before any optimization loop exists.

Acceptance Criteria

skills/evals/triage/cases.jsonl exists; each line is a JSON object pairing a triage input (a pre-fetched PR context as triage.md consumes) with the expected decision fields escalate (bool) and risk (LOW|MEDIUM|HIGH).
A case schema (skills/evals/case.schema.json) plus a README document the required fields, and the schema is validatable the same way scripts/initiative-planner/plan.schema.json is (python jsonschema).
Cases cover both escalate=true and escalate=false outcomes and include at least one case for each high-risk trigger named in triage.md decision criteria (auth/secrets, DB migrations/schema, security anti-patterns).
A CODEOWNERS entry marks skills/evals/ as separately owned so a future proposer can never edit the held-out set (reward-hacking guard).

Tasks / Subtasks

Create skills/evals/ tree and case.schema.json mirroring the plan.schema.json JSON-Schema style (AC: Go-live improvements for PR review agent #2)
- Define required keys: input (PR context), expected.escalate, expected.risk
- additionalProperties:false like plan.schema.json
Author skills/evals/triage/cases.jsonl grounded strictly in triage.md decision criteria (AC: test issue from agent #1, Add @claude delegation, auto-merge, and rebase handling #3)
- Include escalate=false safe cases
- Include escalate=true cases per high-risk area
Write skills/evals/README.md documenting the case format and the separately-owned invariant (AC: Go-live improvements for PR review agent #2, Optimize review: small-PR and incremental fast paths #4)
Add CODEOWNERS entry for skills/evals/ (AC: Optimize review: small-PR and incremental fast paths #4)

Dev Notes

triage.md emits a deterministic JSON object (escalate, risk, signals, summary) with no tools — this is exactly why it is the right first skill: expected outputs can be fixed values. Ground each case in the numbered decision criteria, not invented behavior.
Model the case schema on scripts/initiative-planner/plan.schema.json (draft 2020-12, additionalProperties:false) and validate with the same inline pip install 'jsonschema>=4' pattern CI already uses.
Cases are the trust anchor of the whole initiative. They MUST live separately from any scorer/proposer and be CODEOWNER-gated; the proposer may only ever edit the skill markdown, never the test set.
This story is data + schema + docs only — no scorer and no model invocation. Keep it reviewable in isolation.
Directory naming follows the idea's literal skills/evals/<skill>/cases.jsonl; flagged in open_questions since repo skills live under prompts/ and frameworks/.

Project Structure Notes

Introduces a new top-level skills/evals/ directory per Discussion #572. No existing eval/test-fixture directory covers prompt-skill quality; tests/ holds bats/python for shell + workflow contracts, not prompt evals.

References

prompts/triage.md#Decision-criteria
prompts/shared.md
scripts/initiative-planner/plan.schema.json
scripts/initiative-planner/validate-plan.py
.github/workflows/lint.yml#bats
AGENTS.md

Likely target surface

skills/evals/case.schema.json
skills/evals/triage/cases.jsonl
skills/evals/README.md
CODEOWNERS

Story prepared by the BMAD Scrum Master (Bob) for epic #581. Status: ready-for-dev.

Dev-Lead Implementation Plan

Issue: #582 — [Phase 1] Define held-out eval-case format + seed prompts/triage.md case set

Scope

Introduce a versioned, CODEOWNER-gated held-out eval-case format (skills/evals/) with a JSON-Schema (draft 2020-12, additionalProperties:false) mirroring plan.schema.json, a python jsonschema validator, a seed triage/cases.jsonl grounded strictly in prompts/triage.md decision criteria, and a README documenting the format + separately-owned invariant.

Implementation Checklist

skills/evals/case.schema.json — draft 2020-12, additionalProperties:false; required keys id, input, expected (expected.escalate bool, expected.risk LOW|MEDIUM|HIGH), optional description/tags
skills/evals/validate-cases.py — validate every JSONL line against the schema + enforce unique id (mirrors validate-plan.py / inline pip install 'jsonschema>=4' pattern)
skills/evals/triage/cases.jsonl — seed cases grounded in triage.md numbered criteria, covering escalate=false and escalate=true incl. one per high-risk trigger (auth/secrets, DB migrations/schema, security anti-patterns)
skills/evals/README.md — document case format, validation command, and the separately-owned / reward-hacking-guard invariant
.github/CODEOWNERS — add /skills/evals/ entry (separately owned; proposer can never edit held-out set)
.github/workflows/lint.yml — add the new bats test file to the bats job list (per AGENTS.md repo-specific test-list note)

Tests to Write

tests/test_evals_cases.bats — validator accepts real cases.jsonl; rejects missing expected.risk, unknown property, bad risk enum, duplicate id; asserts coverage of both escalate outcomes and each named high-risk trigger via tags

Test Command

bats tests/test_evals_cases.bats (full suite via the bats job list in .github/workflows/lint.yml); lint via bash .dev-lead/scripts/dev-lead-lint.sh

Dev-Lead: Implementation Complete

Plan Execution

skills/evals/case.schema.json — draft 2020-12, additionalProperties:false; required id/input/expected with expected.escalate (bool) + expected.risk (LOW|MEDIUM|HIGH), optional description/tags (AC test issue from agent #1, Go-live improvements for PR review agent #2)
skills/evals/validate-cases.py — validates every JSONL line against the schema and enforces unique id; mirrors validate-plan.py and the inline pip install 'jsonschema>=4' CI pattern (AC Go-live improvements for PR review agent #2)
skills/evals/triage/cases.jsonl — 8 seed cases grounded strictly in prompts/triage.md numbered criteria (AC test issue from agent #1, Add @claude delegation, auto-merge, and rebase handling #3)
skills/evals/README.md — documents the case format, validation command, and the separately-owned / reward-hacking-guard invariant (AC Go-live improvements for PR review agent #2, Optimize review: small-PR and incremental fast paths #4)
.github/CODEOWNERS — /skills/evals/ entry so the held-out set is separately owned (AC Optimize review: small-PR and incremental fast paths #4)
.github/workflows/lint.yml — added tests/test_evals_cases.bats to the bats list and skills/** to the lint trigger paths so the set is validated in CI
TDD: tests/test_evals_cases.bats written first (red), then implemented to green

Coverage (AC #3)

escalate=false: triage-docs-typo (LOW), triage-test-only (LOW), triage-clean-logic-fix (MEDIUM)
escalate=true: triage-auth-token (HIGH, auth/secrets), triage-db-migration (HIGH, DB migration/schema), triage-sql-concat (HIGH, security anti-pattern), triage-workflow-pr-target (HIGH), triage-unresolved-thread (MEDIUM)
Risk spread LOW/MEDIUM/HIGH; every HIGH case escalates (asserted by a test).

Test Results

1..10
ok 1 validate-cases accepts the seed triage case set
ok 2 validate-cases rejects a case missing expected.risk
ok 3 validate-cases rejects an unknown top-level property
ok 4 validate-cases rejects an invalid risk enum value
ok 5 validate-cases rejects a non-boolean escalate
ok 6 validate-cases rejects duplicate case ids
ok 7 case schema validates with the same python jsonschema pattern as plan.schema.json
ok 8 seed cases cover both escalate outcomes
ok 9 seed cases include a case for each named high-risk trigger
ok 10 every high-risk (HIGH) seed case escalates

Full bats suite (13 files / 206 tests) passes; bash .dev-lead/scripts/dev-lead-lint.sh passes; markdownlint and CI-config yamllint clean on the changed files.

Files Changed

skills/evals/case.schema.json: new JSON-Schema for one eval case
skills/evals/validate-cases.py: new JSONL-vs-schema validator (no scorer/model)
skills/evals/triage/cases.jsonl: 8 seed triage cases
skills/evals/README.md: case-format + separately-owned-invariant docs
.github/CODEOWNERS: separate ownership for /skills/evals/
.github/workflows/lint.yml: wire the new bats test + skills/** lint trigger
tests/test_evals_cases.bats: schema/coverage tests

Notes

Scope kept to data + schema + docs only — no scorer, no model invocation, per the story's Dev Notes.
Directory naming follows the story's literal skills/evals/<skill>/cases.jsonl (the open question noted that repo skills otherwise live under prompts//frameworks/). Flagging for reviewer confirmation.
CODEOWNERS owner set to @petry-projects/org-leads (a known-existing team) to guarantee a valid rule; tighten to a dedicated eval-owners team if/when one exists.

feat: implement issue #582 — [Phase 1] Define held-out eval-case form…

refactor(evals): relocate held-out eval set to repo-root evals/ per m…

feat: implement issue #582 — [Phase 1] Define held-out eval-case form…

feat: implement issue #583 — deterministic eval scorer (triage) over #…

feat: implement issue #582 — [Phase 1] Define held-out eval-case form…

Navigation Menu

Repository navigation

[Phase 1] Define held-out eval-case format + seed prompts/triage.md case set #582

Story

Acceptance Criteria

Tasks / Subtasks

Dev Notes

Project Structure Notes

References

Likely target surface

Dev-Lead Implementation Plan

Scope

Implementation Checklist

Tests to Write

Test Command

Dev-Lead: Implementation Complete

Plan Execution

Coverage (AC #3)

Test Results

Files Changed

Notes

14 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Phase 1] Define held-out eval-case format + seed prompts/triage.md case set #582

Description

Story

Acceptance Criteria

Tasks / Subtasks

Dev Notes

Project Structure Notes

References

Likely target surface

Activity

don-petry commented on Jun 12, 2026

Dev-Lead Implementation Plan

Scope

Implementation Checklist

Tests to Write

Test Command

don-petry commented on Jun 12, 2026

Dev-Lead: Implementation Complete

Plan Execution

Coverage (AC #3)

Test Results

Files Changed

Notes

14 remaining items

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions