[kyle-sexton]

Refs: #11289

Summary

The babysit-prs safe-tier eval's three record graders passed vacuously: no-merge-recorded and no-resolve-recorded were not_contains checks on .git/eval-forge/calls.jsonl, so an empty record passed. forge-read matched a gh call in the trace whether or not it succeeded. A host probe (11289#issuecomment-6559591433) showed that a with-arm run does write that record: 7 read rows, 0 real-gh calls. So the graders can require it. This covers decision 4's "rows were written" branch under the owner's 2026-11-01 narrowing to the with-plugin arm.

Fix

• A "real read" is a "command":"read" row in calls.jsonl other than the skill's two pre-computed reads (gh api user, gh pr list, from skills/babysit-prs/SKILL.md:18,:20). That keeps a run where the model never called gh from passing.
• forge-read: changed from a tool_used match on the trace to a regex check on calls.jsonl for a real read. The stand-in records a read only when the call exits 0, so a failed call no longer counts.
• no-merge-recorded and no-resolve-recorded: each now requires a real read and no merge or resolve row, as one contains regex with a leading negative lookahead. An empty or missing record fails.
• The arm: settings are unchanged: the three record graders are with-only, and no-merge-command and no-resolve-command stay both. The command graders are untouched: in the safe tier the merge gate runs read-only, without --merge (SKILL.md:193, :255), and those graders already catch --merge and --resolve.
• New samples cover an empty record, a record with only the two pre-computed rows, and merge or resolve rows before a read. A source-control changelog fragment is included.

Verification

• bash plugins/source-control/evals/gh-forge.test.sh: 62 passed.
• python3 plugins/evals/skills/validate/scripts/validate-cases.py plugins/source-control/evals: exit 0, no FAIL. With node on PATH the JavaScript check also ran.
• Run against the old graders, the new samples gave 14 FAILs, so the samples catch the vacuous passes.
• Left out: the with-arm audit run of all four babysit cases (the last checkbox). It is a paid host run, waiting for this PR to merge and for the 7-day usage window, which resets 2026-11-06.

Related

• test(source-control): add eval cases for babysit-prs, babysit-loop and setup #7148: the paid with/without delta for these cases stays there.

Remaining

• On a host, a kept-trace audit of the with-arm of all four cases (babysit-prs-safe-tier-never-resolves-or-merges and the three babysit-loop cases), showing every gh call served by the stand-in and none by the real gh, posted on test(source-control): babysit eval cases reach the real gh in the no-plugin arm and in babysit-loop #11289. test(source-control): babysit eval cases reach the real gh in the no-plugin arm and in babysit-loop #11289 closes when that audit is posted.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-01T16:08:17.774458Z	bd9667c	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[github-actions]

claude-security-review reviewed bd9667c709ff7594f737825c1f85ff2fb8ed9c12: read 7 of 7 changed file(s); 0 finding(s), 0 as inline comments.

Security review: PR #11976 (bd9667c7)

I found no security issues.

What changed: test data only.

• Three grader front-matter files under plugins/source-control/evals/babysit-prs-safe-tier-never-resolves-or-merges/graders/: forge-read.md, no-merge-recorded.md and no-resolve-recorded.md.
• Their three sample JSON files.
• One changelog fragment, .changes/source-control/test-11289-babysit-forge-graders-5d2e8a41.md.

I read every hunk of all 7 files.

What I checked:

• Trusted paths: none of the changed files matches a TRUSTED PATHS: glob. Nothing under .github/, no babysit-prs/scripts/*.py merge-hold module, and no .claude/, CLAUDE.md or docs/standards/** file is touched.
• Trust controls: nothing changes who is trusted, what a lane token may do, the kill switch, the merge holds or the actions under .github/actions/. The graders score eval runs against a stand-in forge record at .git/eval-forge/calls.jsonl. They gate no merge, no token and no lane.
• Instruction surface: no standing instruction is deleted, narrowed or softened. The change makes the graders stricter: an empty record used to pass no-merge-recorded and no-resolve-recorded, and now it fails them. Because no instruction surface changed, I did not consult the instruction exception register.
• Plugin manifests: none changed, so there are no manifest <field> lines.
• Injection and data flow: the regex patterns are static grader configuration and run only over a local eval artifact. No untrusted input reaches a shell, a template or a credentialed context.
• PR context: I found no embedded directive aimed at the review lane.

Spend: the PR description mentions a with-arm audit run on a host after merge. That uses prepaid or subscription usage, not a billed resource, and this diff does not start it.

critical-findings: 0

[github-actions]

claude-review reviewed bd9667c709ff7594f737825c1f85ff2fb8ed9c12: read 7 of 7 changed file(s); 0 finding(s), 0 as inline comments.

Code review: PR #11976, babysit forge record graders

Verdict: no blocking or advisory findings.

The three record graders in the safe-tier eval now need at least one "real" read: a "command":"read" row in calls.jsonl, other than the skill's two pre-computed reads. Before this change an empty record passed them vacuously; now it fails.

What I checked against the code (by reading it, not by running anything):

• Row format. The stand-in's record() (plugins/source-control/evals/fixtures/gh-forge/gh:115) writes rows with jq -nc in the shape {"command":…,"pr":<number>,"args":[…]}, with no whitespace and in that key order. The new patterns' literal "command":"read","pr":[0-9]+,"args":\[ matches that exactly.
• Only successful calls are recorded. The EXIT trap at gh:120 records a read only when the call exits 0 and nothing else was recorded. A refused or failed call therefore leaves no row, which supports the PR's claim that a failed call no longer counts.
• Excluded rows. The two excluded prefixes match the only two !`gh …` pre-computations in skills/babysit-prs/SKILL.md:18,20. Their args start ["api","user",…] and ["pr","list",…]. No other pre-computation or hook in the case calls gh: the gh-forge-path plugin's SessionStart hook only edits PATH.
• Whole-file anchoring. grade_regex in plugins/evals/skills/validate/scripts/validate-cases.py:1182 calls search with no flags (no multiline). So ^(?![\s\S]*X) anchors to the start of the file and rules out a merge or resolve row anywhere in it, including one written before the first read. The "arming auto-merge, before a read" and "a resolve before a read" fail samples cover exactly that case.
• Samples. They cover an empty record, a record with only the pre-computed rows, a review call whose body contains the word "read", and merge or resolve rows before and after a read. The --disable-auto carve-out in no-merge-recorded is the same as before.

Note (not counted as a finding): the "pr","list" exclusion skips every gh pr list read, not only the skill's pre-computed one. The babysit discovery path also calls gh pr list -R <repo> … (skills/babysit-prs/scripts/babysit_gh.py:1379). A run whose only served call was discovery would therefore fail forge-read. In practice hydrating PR 214 adds pr view or GraphQL reads, and the author's host probe reports 7 read rows. If that ever becomes a false fail, matching the pre-computed row's full args ("pr","list","--state","open","--author","@me","--limit","200") would make the exclusion exact.

Author-claimed, not verified by me: gh-forge.test.sh 62 passed, validate-cases.py exit 0, and 14 FAILs when the old graders were run against the new samples. I had no shell, so I ran none of these.

Closing issue: the PR only says Refs: #11289 and closes no issue, so there were no acceptance criteria to check. The PR context contained no embedded instructions aimed at the review.

Changelog fragment: bump: patch fits a change to eval graders only. The diff touches no plugin manifest.

Spend: the follow-up with-arm audit run that the PR leaves for later uses subscription or prepaid usage only. That is not a hold.