Skip to content

fix(evals): step6 cases scaffold their history, prompts stop leaking the answer (#389) - #404

Merged
wasikarn merged 1 commit into
developfrom
claude/eval-tells-389
Oct 3, 2026
Merged

wasikarn merged 1 commit into
developfrom
claude/eval-tells-389

Conversation

@wasikarn

@wasikarn wasikarn commented Oct 3, 2026

Copy link
Copy Markdown
Owner

Closes #389.

What changed

  • Test first. tests/evals/test-eval-cases.sh scans every prompt body (frontmatter stripped) for planted|rubric|grader|no bug(s)|accurate|correctly implemented. No per-case allowlist. Run before the fix, it flagged exactly deep-audit-step6-zero-fixes-skip/prompt.md lines 15, 16, 55, 56. Nothing else under evals/ was flagged. A negative control checks that a clean prompt passes and the same prompt with one tell appended fails.
  • Scaffold move. All six deep-audit-step6-* cases now have a case.yaml and an executable scaffold.sh, following the deep-audit-planted pattern. The scaffold builds the same two-commit history the prompt used to dictate (same files, same commit messages, and NOTES.md committed in commit 2). Each prompt now only states the task: the latest commit is the session, the earlier one is pre-existing, the claims are that commit's message plus NOTES.md, and the test command.
  • Paraphrased tells removed too. The regex can't catch these, so they went out by hand: "nothing else imports/calls/reads" (single-local-fix-skip), "two small, unrelated modules" and "share no code, no imports, no data" (two-fixes-clean), "exactly one new area" (single-nonlocal-trap), and "test_helpers.py is not touched" (shared-test-fixture). Graders are unchanged. Each one still judges the same fixture.
  • evals/README.md: the rule ("a prompt states the task, never the expected verdict or the trigger fact; fixture history goes in scaffold.sh"), the limit of the word-list check, and a tradeoff note. Without the tell, zero-fixes-skip now also counts false "fixes", which overlaps deep-audit-clean. The c182ffc6 pilot's two should-not-fire results were measured with the tells in place.

Verification

  • Mutation checks: putting a tell line back makes the case fail. Making the scan always exit 0 makes the negative control fail.
  • Each scaffold.sh was run in a fresh temp dir. Every case gets its own repo with 2 commits and a clean tree. The suite fails as each grader expects (zero-fixes-skip passes; the others fail 1 or 2 tests). It also works when the dir sits under a parent .git.
  • audit.sh: 0 CRIT. run-gauntlet.sh (GAUNTLET_JOBS=2): all layers passed.

Not confirmed: git init in the live runner

I did not do a paid live run. The scaffold's git init -q . follows a pattern already in use: 14 existing scaffold.sh files do it, and the README records that deep-audit-clean (a git-history scaffold) was checked against the real CLI. The installed CLI's own text says --scaffold runs the script as the operator, before the subject's sandboxed session starts. The old prompts' "Operation not permitted" note was about the subject running git init inside that sandbox, which is a different context. This is confirmation by precedent, not by a run.

Follow-up for the operator (paid): one --scaffold --keep-temp run of a step6 case to confirm the scaffold, then re-run the step6 ablation. The earlier pilot numbers no longer apply to these cases.

…the answer (GH #389)

The six deep-audit-step6 cases had the subject type the fixture history
from the prompt, and two prompts told it the expected answer
(zero-fixes-skip: 'no bugs', 'accurate', 'no planted bug';
single-local-fix-skip: 'nothing else imports'). Each case now has a
scaffold.sh + case.yaml (deep-audit-planted pattern) that builds the
two-commit history, and the prompt only states the task.

test-eval-cases.sh gains a body-only deny-list scan with a negative
control; before this change it flagged exactly zero-fixes-skip lines
15, 16, 55, 56 and nothing else under evals/.
@wasikarn
wasikarn merged commit 8d527a7 into develop Oct 3, 2026
3 checks passed
@wasikarn
wasikarn deleted the claude/eval-tells-389 branch October 3, 2026 15:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

evals: prompts leak the expected answer; step6 cases have no scaffold

1 participant