Skip to content

[Phase 1] Spec-drift frozen offline eval + tune to zero false positives #1145

Closed
#1225
Closed
[Phase 1] Spec-drift frozen offline eval + tune to zero false positives#1145
#1225
Labels
dev-leadFor dev-lead agent pickupinitiativeEpic / initiative tracking issue

Description

@github-actions
github-actions
bot
opened on Jul 10, 2026

Story

As a maintainer who must trust the drift detector before wiring it,
I want a frozen offline eval with dev and held-out splits that tunes the detector on dev only and proves zero false positives on an immutable holdout set,
so that the detector is validated against reward-hacking and stays honest — its baseline can't be quietly moved to make a bad detector look good.

Acceptance Criteria

  1. A frozen offline eval lives under evals/spec-drift/ with a dev and a held-out holdout split (mirroring evals/deep-review/), each case a labeled (diff, acceptance-criteria, expected-verdict) triple conforming to a documented case shape.
  2. The eval harness and its scoring are runnable offline (no live PR required) and documented in evals/spec-drift/README.md; it drives the Story-2 detector's pure classifier directly.
  3. Overfitting / reward-hacking guard: the detector is tuned ONLY against evals/spec-drift/dev; the holdout split is never read during tuning.
  4. Artifact-immutability guard: the holdout cases and their expected-verdict baseline are committed as frozen fixtures — changing any holdout expectation requires an explicit, reviewed fixture change, not a tuning-loop side effect.
  5. The tuned detector scores 0 false positives on the holdout split, and this is demonstrated by the offline harness.

Tasks / Subtasks

Dev Notes

  • This story depends on the Story-2 detector and validates it; it changes no production behavior — it adds an eval directory and its harness.
  • Follow the split pattern already established by evals/deep-review/holdout (a dev set you tune on and a holdout set you never train against) and the case-schema conventions in evals/case.schema.json. The frozen holdout baseline is the artifact-immutability guard.
  • Reward-hacking guard is the whole point: the tuning loop must physically only read dev. Document in the README that promoting a detector change requires holdout still at 0 false positives, and that editing a holdout expectation is a reviewed fixture change, never automatic.
  • Keep the eval offline and deterministic where possible (fixture diffs + fixture ACs), so it runs without a live PR and can gate future detector changes.
  • Testing standard: the harness itself should be lint-clean; if it includes shell helpers, shellcheck them and add bats coverage consistent with the repo's eval tooling (evals/validate-cases.py).

Project Structure Notes

Adds evals/spec-drift/ (dev + holdout + README + harness). Mirrors evals/deep-review/; no existing files change.

References

  • evals/deep-review/holdout
  • evals/case.schema.json
  • evals/validate-cases.py
  • evals/README.md
  • AGENTS.md

Likely target surface

  • evals/

Story prepared by the BMAD Scrum Master (Bob) for epic #1142. Status: ready-for-dev.

Activity

  1. github-actions
    added
    initiativeEpic / initiative tracking issue
    on Jul 10, 2026
  2. github-actions
    added a parent issue on Jul 10, 2026
  3. github-actions
    marked this as blocked on Jul 10, 2026
  4. github-actions
    marked this as blocking on Jul 10, 2026
  5. don-petry
    added
    dev-leadFor dev-lead agent pickup
    on Jul 14, 2026

  6. don-petry commented on Jul 14, 2026

    @don-petry
    don-petry
    on Jul 14, 2026
    Collaborator

    Dev-Lead Implementation Plan

    Issue: #1145 — [Phase 1] Spec-drift frozen offline eval + tune to zero false positives

    Scope

    Add a frozen, offline, deterministic eval under evals/spec-drift/ (dev + holdout splits, README, case schema, harness) that drives the Story-2 (scripts/spec-drift.sh) pure classifier classify_drift directly, is tuned only on dev, and proves 0 false positives on an immutable holdout split. No production behavior changes.

    Design notes

    • Mirrors evals/deep-review/ (dev/holdout) and follows evals/case.schema.json + evals/validate-cases.py split-hygiene conventions.
    • The deterministic detector surface is the pure classify_drift(analysis) function. Each case is a labeled (diff, acceptance-criteria, expected-verdict) triple and carries a frozen cheap-tier detector analysis fixture — the same "embed the upstream tier's output as a fixture" pattern evals/deep-review uses (its embedded Triage result:), so scoring needs no live PR or LLM.
    • "False positive" = classifier emits DRIFT when the expected verdict is not DRIFT (flags a spec-compliant PR). Holdout includes FP-trap cases (prose mentioning "drift" but verdict ALIGNED, empty/timeout → INDETERMINATE) plus genuine DRIFT true-positives.
    • Reward-hacking guard: the harness physically reads only the split it is given (run-eval.sh dev never opens holdout/). Holdout cases + expected verdicts are committed as frozen fixtures, already CI-immutable via holdout-guard.yml / CODEOWNERS over evals/**/holdout/.

    Implementation Checklist

    • evals/spec-drift/case.schema.json — documented case shape (JSON-Schema 2020-12).
    • evals/spec-drift/dev/cases.jsonl — proposer-visible tuning cases.
    • evals/spec-drift/holdout/cases.jsonl — frozen held-out cases (0-FP baseline).
    • evals/spec-drift/run-eval.sh — offline harness: sources scripts/spec-drift.sh, runs classify_drift per case, scores, fails on any false positive.
    • evals/spec-drift/README.md — case shape, how to run offline, reward-hacking + artifact-immutability guards.
    • Add tests/test_spec_drift_eval.bats to the bats job list in .github/workflows/lint.yml.

    Tests to Write

    • tests/test_spec_drift_eval.bats — pure scoring helpers (false-positive detection), end-to-end harness run over a temp fixture dir, and split-isolation (holdout not read when scoring dev).
    • Schema conformance of both splits via evals/validate-cases.py <cases.jsonl> evals/spec-drift/case.schema.json.
    • Split hygiene via evals/validate-cases.py (dev/holdout disjoint ids).

    Test Command

    bats tests/test_spec_drift_eval.bats tests/test_spec_drift.bats tests/test_validate_cases.bats and bash evals/spec-drift/run-eval.sh holdout

  7. don-petry commented on Jul 14, 2026

    @don-petry
    don-petry
    on Jul 14, 2026
    Collaborator

    Dev-Lead: Implementation Complete

    Plan Execution

    • Added evals/spec-drift/case.schema.json — documented case shape (JSON-Schema 2020-12), following evals/case.schema.json conventions.
    • Added evals/spec-drift/dev/cases.jsonl (7 proposer-visible tuning cases) and evals/spec-drift/holdout/cases.jsonl (8 frozen held-out cases).
    • Added evals/spec-drift/run-eval.sh — offline harness that sources scripts/spec-drift.sh and drives its pure classify_drift directly, scoring each case's frozen analysis fixture against its expected verdict.
    • Added evals/spec-drift/README.md — case shape, offline run instructions, reward-hacking guard (tune on dev only), and artifact-immutability guard (frozen holdout).
    • Added tests/test_spec_drift_eval.bats and wired it into the bats job in .github/workflows/lint.yml.
    • Confirmed the classifier already scores 0 false positives on both splits with no change to scripts/spec-drift.sh — no production behavior changed.

    Acceptance Criteria

    • AC1 — evals/spec-drift/ has dev + holdout splits mirroring evals/deep-review/; each case is a labeled (diff, acceptance-criteria, expected-verdict) triple conforming to case.schema.json. ✔
    • AC2 — Harness + scoring run fully offline (no live PR/LLM); it drives the Story-2 pure classifier classify_drift directly; documented in README.md. ✔
    • AC3 — The harness physically reads only the split(s) named on the CLI (run-eval.sh dev never opens holdout/); proven by a bats test using a dev-only root. ✔
    • AC4 — Holdout cases + expected-verdict baseline are committed as frozen fixtures; immutability enforced by CODEOWNERS /evals/**/holdout/ and holdout-guard.yml (the evals/ prefix already covers this path). ✔
    • AC5 — The detector scores 0 false positives on holdout, demonstrated by the harness. ✔

    Test Results

    $ bash evals/spec-drift/run-eval.sh
SUMMARY split=dev total=7 correct=7 false_positives=0 false_negatives=0 mismatches=0
SUMMARY split=holdout total=8 correct=8 false_positives=0 false_negatives=0 mismatches=0

$ bats tests/test_spec_drift_eval.bats
1..13
ok 1 sd_is_false_positive: DRIFT actual vs ALIGNED expected is a false positive
...
ok 13 main: both committed splits validate against the documented case schema

$ bats tests/test_spec_drift.bats tests/test_validate_cases.bats tests/test_holdout_guard.bats
# all pass (63 tests), incl. "validate-cases validates the committed evals tree"

$ python3 evals/validate-cases.py
OK: 5 skill(s), 46 case(s) across dev+holdout, no cross-split id overlap.

$ bash .dev-lead/scripts/dev-lead-lint.sh
  [lint] all checks passed

$ shellcheck --severity=warning -x evals/spec-drift/run-eval.sh
# clean

# Negative control (injected false positive) — the gate correctly fails:
  FAIL  inject-fp  expected=ALIGNED actual=DRIFT (FALSE POSITIVE)
::error::[spec-drift-eval] one or more splits had a false positive ...  (exit 1)

    Files Changed

    • evals/spec-drift/case.schema.json (new): documented case shape.
    • evals/spec-drift/dev/cases.jsonl (new): 7 tuning cases (token variants, timeout, both-tokens, an FP-trap).
    • evals/spec-drift/holdout/cases.jsonl (new): 8 frozen cases (FP-traps that mention "drift"/"no drift"/timeout but are ALIGNED/INDETERMINATE, plus DRIFT true-positives).
    • evals/spec-drift/run-eval.sh (new): offline harness; gate fails if any scored split has a false positive.
    • evals/spec-drift/README.md (new): usage + reward-hacking/immutability guards.
    • tests/test_spec_drift_eval.bats (new): pure-helper, per-split, end-to-end, split-isolation, and schema-conformance coverage.
    • .github/workflows/lint.yml: added tests/test_spec_drift_eval.bats to the bats job (per AGENTS.md's repo-specific bats-list extension).

    Notes

    • A frozen analysis fixture per case makes offline/deterministic scoring possible: classify_drift maps the cheap-tier analysis text to a verdict, so each case records that text — the same "embed the upstream tier's output as a fixture" pattern evals/deep-review/ uses (its Triage result:). The diff + acceptance_criteria ground the scenario; scoring reads analysis.
    • "False positive" is defined as emitting DRIFT when the expected verdict is not DRIFT. Missed drift is a false negative and does not fail the gate — matching this story's explicit "zero false positives" scope.
    • The CI ShellCheck job is scoped to scripts/**, so it does not cover evals/spec-drift/run-eval.sh; the harness is shellcheck-clean (verified locally) and its behavior is guarded by tests/test_spec_drift_eval.bats in the bats job.
    • No follow-ups required for this story.
  8. don-petry
    added a commit that references this issue on Jul 14, 2026

    feat: implement issue #1145 — [Phase 1] Spec-drift frozen offline eva…

    190610b
  9. don-petry
    closed this as completedin #1225on Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    dev-leadFor dev-lead agent pickupinitiativeEpic / initiative tracking issue

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions