Repository navigation
fix(evals): step6 cases scaffold their history, prompts stop leaking the answer (#389) - #404
Merged
Merged
Conversation
…the answer (GH #389) The six deep-audit-step6 cases had the subject type the fixture history from the prompt, and two prompts told it the expected answer (zero-fixes-skip: 'no bugs', 'accurate', 'no planted bug'; single-local-fix-skip: 'nothing else imports'). Each case now has a scaffold.sh + case.yaml (deep-audit-planted pattern) that builds the two-commit history, and the prompt only states the task. test-eval-cases.sh gains a body-only deny-list scan with a negative control; before this change it flagged exactly zero-fixes-skip lines 15, 16, 55, 56 and nothing else under evals/.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #389.
What changed
tests/evals/test-eval-cases.shscans every prompt body (frontmatter stripped) forplanted|rubric|grader|no bug(s)|accurate|correctly implemented. No per-case allowlist. Run before the fix, it flagged exactlydeep-audit-step6-zero-fixes-skip/prompt.mdlines 15, 16, 55, 56. Nothing else underevals/was flagged. A negative control checks that a clean prompt passes and the same prompt with one tell appended fails.deep-audit-step6-*cases now have acase.yamland an executablescaffold.sh, following thedeep-audit-plantedpattern. The scaffold builds the same two-commit history the prompt used to dictate (same files, same commit messages, andNOTES.mdcommitted in commit 2). Each prompt now only states the task: the latest commit is the session, the earlier one is pre-existing, the claims are that commit's message plusNOTES.md, and the test command.evals/README.md: the rule ("a prompt states the task, never the expected verdict or the trigger fact; fixture history goes inscaffold.sh"), the limit of the word-list check, and a tradeoff note. Without the tell,zero-fixes-skipnow also counts false "fixes", which overlapsdeep-audit-clean. Thec182ffc6pilot's two should-not-fire results were measured with the tells in place.Verification
scaffold.shwas run in a fresh temp dir. Every case gets its own repo with 2 commits and a clean tree. The suite fails as each grader expects (zero-fixes-skip passes; the others fail 1 or 2 tests). It also works when the dir sits under a parent.git.audit.sh: 0 CRIT.run-gauntlet.sh(GAUNTLET_JOBS=2): all layers passed.Not confirmed:
git initin the live runnerI did not do a paid live run. The scaffold's
git init -q .follows a pattern already in use: 14 existingscaffold.shfiles do it, and the README records thatdeep-audit-clean(a git-history scaffold) was checked against the real CLI. The installed CLI's own text says--scaffoldruns the script as the operator, before the subject's sandboxed session starts. The old prompts' "Operation not permitted" note was about the subject runninggit initinside that sandbox, which is a different context. This is confirmation by precedent, not by a run.Follow-up for the operator (paid): one
--scaffold --keep-temprun of a step6 case to confirm the scaffold, then re-run the step6 ablation. The earlier pilot numbers no longer apply to these cases.