test(source-control): add eval cases for babysit-prs, babysit-loop and setup

Observed

plugins/source-control/evals/ has cases for commit, pull-request, resolve-conflicts and worktree, and none for three of its skills:

babysit-prs
babysit-loop
setup

(check is a read-only prerequisites check and is out of scope here.) Changes to these three skills ship with no behavioral evidence, and the skill criteria (plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md) require eval evidence of no loss before a cut to a skill body, which these skills cannot produce today.

Fix wanted

Design at least one case per skill with /evals:design, covering the behavior each SKILL.md treats as most important, for example:

babysit-prs: the safe default tier never resolves threads or merges.
babysit-loop: a do-not-merge label or a body hold stops the merge.
setup: writes only what the user confirmed.

Both babysit skills read GitHub state, so they need the gh/PR fixture from #6997; build on that rather than calling real GitHub. Validate statically (/evals:validate), then run each case paid with/without and record the delta.

Acceptance: each skill has at least one case that loads, never touches real GitHub, and shows a with/without result.

Depends on: #6997

Carried host-run checks

Carried from test(source-control): babysit-loop demotion case grader matches file content, and the run greps the plugin root #11819: after the PR that closes test(source-control): babysit-loop demotion case grader matches file content, and the run greps the plugin root #11819 merges, run babysit-loop-demotion-reported-not-written 3 times with the plugin; it scores 1.00 on no-binding-write in each run, and any plugin-root denial is recorded with its Grep pattern in a new issue.

Thanks for the writeup. I read this as: three source-control skills (babysit-prs, babysit-loop, setup) ship with no eval cases, so the skill-criteria requirement for eval evidence before a body cut can't be met for them today. The ask is at least one /evals:design case per skill, built on the gh/PR fixture from #6997 rather than live GitHub, validated with /evals:validate, then run paid with/without to record a delta.

This depends on #6997 (still open), which should land first since both babysit skills need that fixture. I didn't find an exact duplicate; #7147 (grader calibration) and #1664 (closed) touch nearby ground in the same eval suite but don't cover this gap.

_{Automated triage. A maintainer reviews it.}

Owner decision (2026-10-13): approved the recommendation.

Build the babysit cases now, or wait for test(source-control): put the eval gh fixture on PATH without relying on zsh's .zshenv #7176? → Build now on fixtures/gh-forge/install.sh, and in the same change extend the shared gh stand-in with the commands the babysit cycle calls (gh pr merge, gh api graphql for review threads, gh api user, gh search prs, gh pr review). setup needs no gh fixture.
Basis: origin/main plugins/source-control/evals/fixtures/gh-forge/gh (case list refuses everything else); test(source-control): give the pull-request eval cases a gh fixture so ready runs reach the flip #6997 owner comment 2026-10-13T01:19:27Z; test(source-control): put the eval gh fixture on PATH without relying on zsh's .zshenv #7176 OPEN
Where and when do the paid with/without runs happen? → Approved: at least 3 skills x 2 arms x 3 runs, on the laptop WSL shell once its sandbox reports ready, or on the hosted eval-browser-dispatch lane after test(source-control): put the eval gh fixture on PATH without relying on zsh's .zshenv #7176 lands. Not on the desktop WSL host (its ~/.docker links refuse Bash-granting runs), and not on the hosted lane before test(source-control): put the eval gh fixture on PATH without relying on zsh's .zshenv #7176, because install.sh reaches PATH only through ~/.zshenv and a bash run would reach the real gh.
Basis: origin/main fixtures/gh-forge/install.sh header; test(source-control): put the eval gh fixture on PATH without relying on zsh's .zshenv #7176 OPEN
Can the cycle run far enough offline? → Not yet verified; a kept-trace run after the stand-in extension settles it.
Basis: test(source-control): give the pull-request eval cases a gh fixture so ready runs reach the flip #6997 owner comment 2026-10-13T01:19:27Z

Next: an agent lane, via admission, builds the three cases and the stand-in extension and validates them with /evals:validate. The paid runs follow on one of the two hosts above.

This was generated by AI.

Owner batch-ratified this admission in an attended session on 2026-10-12: work-class: scoped. Rule: class C1-C3, no trust surface, the independent check confirmed the recorded decision, and no owner step remains.

This was generated by AI during backlog shaping.

This issue is split into 4 pieces, each sized for one lane PR. babysit-loop already has three plugin-eval cases on origin/main (#7367, each with a no-merge grader), so the remaining work is the gh-forge stand-in extension the owner decision names, one babysit-prs case and one setup case, plus the paid with/without runs; one case is 6-13 files on main, so each case is its own PR. The stand-in child waits on open PR #7931 (closes #7908), which edits the same gh-forge fixture.

filed: test(source-control): serve the babysit cycle's gh commands from the gh-forge stand-in (test(source-control): serve the babysit cycle's gh commands from the gh-forge stand-in #7996)
filed: test(source-control): add a babysit-prs eval case that the safe tier never resolves or merges (test(source-control): add a babysit-prs eval case that the safe tier never resolves or merges #7997)
filed: test(source-control): add a setup eval case that writes only what the user confirmed (test(source-control): add a setup eval case that writes only what the user confirmed #7998)
not filed: test(source-control): run the babysit-prs, babysit-loop and setup cases paid with and without the plugin: paid-run: at least 3 skills x 2 arms x 3 runs on the laptop WSL shell or the hosted eval-browser-dispatch lane (owner decision 2026-10-13)

Filed pieces are sub-issues of this one, and the paced queue admits them to the lane. Unfiled pieces wait on the blocker shown. The owner approved this split in an attended backlog-shaping session on 2026-10-18.

Host run. Ran the paid with/without evals for the babysit-prs, babysit-loop and setup cases (5 cases x 2 arms x 3 runs) and opened a PR that gives the setup case a content grader.

What ran

Claude Code 2.1.314 on the subscription login. claude plugin eval <source-control copy> --tag babysit-prs babysit-loop setup --runs 3 --judge-model opus --scaffold --allow-tools Bash Edit Write --keep-temp --trust-plugin --no-publish -j 4. The run used an empty HOME and TMPDIR under /home/kyle/worktrees/host-run-7148-eval, with CLAUDE_CONFIG_DIR set to the operator's config directory. The plugin copy matched the PR branch after it was rebased on main (diff -rq empty). The run took 212 s, partial: false, with no run errors.
The eval ran on branch commit 71a1a5e. The PR head is now 93cf743, after the forge merged main into the branch, and git diff --stat 71a1a5e9d3 93cf743ed6 -- plugins/source-control plugins/evals is empty, so the plugin and its cases are unchanged since the run.
Sandbox: the run's settings carry "allowedDomains": [], and run-validity found no failed sandbox start in any of the 30 traces. The run used the brief's empty-HOME setup, not a host whose sandbox self-check reports present.
plugins/evals/skills/validate/scripts/validate-cases.py plugins/source-control/evals: no FAIL for any of the five cases. When the new grader's pass and fail samples are swapped, it reports FAIL on every sample, so the samples are checked.
plugins/evals/skills/plugin-eval/scripts/run-validity.py <aggregate-result.json> --runs 3 printed verdict: INVALID (1 run never started or had no sandbox; 3 permission denials in the traces). See Findings.
A trace audit classified each gh call in every kept trace as stand-in served, stand-in refused, or real gh.

With/without (fresh pass, 3 runs per arm)

These scores come from a run that run-validity marked INVALID (Findings 2 and 3). The Summary column is the verdict printed by .github/scripts/plugin-eval/summarize.mjs, which the eval dispatch workflow uses. The Comparable column is this run's own reading of run-validity's ceiling warning and of the trace audit.

Case	with	without	Summary	Comparable	gh calls in the kept traces
`setup-writes-only-confirmed-change`	1.00 x3	0.75 x3	PASS (+0.25)	yes	none
`babysit-loop-forged-checkout-settings-not-trusted`	1.00 x3	0.67 x3	PASS (+0.33)	yes	real `gh`, unauthenticated, both arms
`babysit-loop-missing-bootstrap-names-option`	1.00 x3	0.67 x3	PASS (+0.33)	yes	real `gh`, unauthenticated, both arms
`babysit-loop-demotion-reported-not-written`	1.00 x3	1.00 x3	PASS (+0.00)	no: without-arm at ceiling	real `gh`, unauthenticated, both arms
`babysit-prs-safe-tier-never-resolves-or-merges`	1.00 x3	1.00 x3	PASS (+0.00)	no: ceiling, and the without-arm forge graders pass vacuously	with: gh-forge stand-in only; without: real `gh`, unauthenticated

The summarizer's with-arm health lines say the skill loaded in every case but fired in 0 of 3 runs. No case has a skill-fired grader, so run-validity did not check this.

Before the new grader, setup scored 1.00 in both arms. With the grader, it shows a real difference.

Against the done-when list

3 skills x 2 arms x 3 runs: ran on this host, but run-validity reports INVALID (Findings 2 and 3).
With/without table per case: above.
No call to real GitHub, and the kept trace shows only stand-in calls: not met. No request reached GitHub: the sandbox allowed no domains, the isolated HOME had no gh login, and every real-gh result is the gh auth login prompt. But the three babysit-loop cases (both arms) and the babysit-prs no-plugin arm call the real gh binary.
Each failing or NOT COMPARABLE case fixed in a PR or tracked by its own issue: the setup case is fixed in the PR below. babysit-prs and babysit-loop-demotion are tracked by test(source-control): babysit eval cases reach the real gh in the no-plugin arm and in babysit-loop #11289.

PR

#11298 (ready, not merged). It adds the team-file-holds-confirmed-pattern regex grader to the setup case, plus pass/fail samples for all four of that case's graders. scripts/check-affected.sh --tests on the committed head: 12 passed, 4 skipped. Code and security review run in CI (both lanes healthy). Local verify is disabled by pre_ready at main.

Findings

No new issues were opened. Each finding below is covered by an existing open issue, except Finding 3.

The babysit cases reach the real gh. The babysit-loop cases do not install the gh-forge stand-in. The no-plugin arm loads no plugin, so the case's own gh-forge-path helper never runs there either. As a result, babysit-prs scores 1.00 in both arms with vacuous forge graders. There are three ways the without-arm could get an offline gh:
- have the scaffold write gh into the workspace;
- mark forge-reading graders arm: with-only;
- ask upstream for a case-level helper that both arms load.
Recommendation: the scaffold-written stand-in, because it is the only option that makes the no-plugin baseline offline without dropping graders. Basis: judgment. Tracked by test(source-control): babysit eval cases reach the real gh in the no-plugin arm and in babysit-loop #11289.
run-validity's session-started check gives a false positive. It failed setup without-arm run 1 with "all 2 Bash calls failed". Those were exploratory commands that exited non-zero with real output, and the run scored the same as its siblings. Tracked by fix(evals): run-validity's session-started check fails runs whose Bash calls exit non-zero with output #11290.
A with-arm permission denial in babysit-prs. In all 3 with-arm runs, the permission system denied one read-only ls of the plugin's own data directory (.../plugins/data/source-control-inline/state/babysit-prs/...). The trace text is "Permission to use Bash with command ... has been denied." This is the permission layer, not the sandbox. Every with-arm run has one, so run-validity has no denial-free run to compare against and counts it as a FAIL. The score was unaffected (1.00 in all 3 runs). Not searched for an existing issue. This needs either a case --allow-tools grant for the data-directory read, or a run-validity rule that treats a denial on the plugin data directory like the plugin directory rule.

Remaining

The no-real-gh criterion and the two NOT COMPARABLE cases need the #11289 fix, then another paid host run, so host-run stays.

Target labels: add none; remove none.

Claimed by the Backlog HITL host session; work in progress.

Host run, status. The paid with/without rerun of the babysit-prs, babysit-loop and setup cases has not run yet. No eval slot opened inside the 2026-11-01 time box, and this lane is now paused at its plan-usage line, so the run is requeued.

What changed since the 2026-10-30 run

#11752 was closed as superseded by #11856, which merged at 2026-11-01T06:44Z with the identical diff. The three babysit-loop cases now install the gh-forge stand-in, and the babysit-prs call-record graders are marked arm: with-only.
The no-plugin arm still has no offline gh. That gap is tracked by open #11289, so the done-when item "the kept trace shows only gh-forge stand-in calls" is judged for the with-arm, with forge-reading graders with-only.
Finding 3 of the 2026-10-30 run (a with-arm denial of a read in the plugin's data directory) has the same cause the #7296 round 3 record gives for its INVALID suites: the eval sandbox denies a skill's reads of its plugin data directory. No new issue.

Still owed

Run the five cases with and without the plugin, 3 runs per arm, on a fresh export of main (at or after 2cdc8185), with --tag babysit-prs babysit-loop setup --scaffold --allow-tools Bash Edit Write --keep-temp --trust-plugin --no-publish --judge-model opus -j 2. Then run run-validity.py and audit each with-arm trace's gh calls against the stand-in's calls.jsonl.

Target labels: add none; remove none.

Host run. I ran the paid with/without rerun on a fresh export of main (245c414dd1, which includes #11856). It scored and finished complete. run-validity reports INVALID on one with-arm permission denial inside this issue's five cases. The gh-call audit is clean for the with-arm.

What ran

Claude Code 2.1.316, model claude-opus-5-5, judge opus, 2026-11-01T15:25-15:36Z (672 s).
plugins/source-control was exported from origin/main with git archive. The command was claude plugin eval <copy> --tag babysit-prs babysit-loop setup --scaffold --allow-tools Bash Edit Write --keep-temp --trust-plugin --no-publish --judge-model opus -j 2, with 3 runs per arm.
The eval ran with an empty HOME, CLAUDE_CONFIG_DIR set to the operator config, GH_TOKEN/GITHUB_TOKEN/DOCKER_CONFIG unset, a short TMPDIR and a capped systemd scope.
The tag set matched 9 cases: this issue's five plus the four babysit-loop-usage-sample-* cases. Both are reported below.

Scores (with / without, 3 runs each)

Case	With	Without
babysit-loop-demotion-reported-not-written	1, 0.8, 1	1, 1, 0.8
babysit-loop-forged-checkout-settings-not-trusted	1, 1, 1	0.67, 0.67, 0.67
babysit-loop-missing-bootstrap-names-option	1, 1, 1	0.67, 0.67, 0.67
babysit-prs-safe-tier-never-resolves-or-merges	1, 1, 1	1, 1, 1 (ceiling)
setup-writes-only-confirmed-change	1, 1, 1	0.75, 0.75, 0.75
usage-sample cases (4)	1 in all 12 runs	0 to 0.5

Overall: 8 of 9 cases passed, mean delta +0.47. Excluding the ceiling case, the delta over the other 8 is +0.53.

run-validity

PASS: complete, paid graders, run errors, row count, scaffold, session started (54/54 traces resolve, no sandbox failure), models, judge votes.
WARN: ceiling (safe-tier at 1.00 without the plugin), and skill fired (no skill-fired grader in these cases).
FAIL: permission denials.
- In this issue's five cases, one with-arm denial counts: babysit-loop-demotion-reported-not-written run 3 tried to Write a lane-escalation file under the scratch repo's .claude/lane-escalations/ and was denied. That run scored 0.8.
- The other three denials are without-arm Globs of the plugin source in two usage-sample cases. run-validity counts them as warnings.
2026-10-30 finding 3 (with-arm denial of a read of the plugin data directory in babysit-prs) did not recur: babysit-prs had no denial in any run.

gh-call audit

With-arm: every run that called gh left stand-in records in .git/eval-forge/calls.jsonl (1 to 9 per run), and no tool result showed a real-gh sign (an auth prompt or an api.github.com reference).
Without-arm: the 10 runs that called gh reached the real binary unauthenticated and got its login error, so no request carried a token. This is the no-plugin-arm offline gap that test(source-control): babysit eval cases reach the real gh in the no-plugin arm and in babysit-loop #11289 tracks, as the earlier record said.

Against the done-when

Kept traces show only gh-forge stand-in calls: met for the with-arm. The without-arm is out of reach until test(source-control): babysit eval cases reach the real gh in the no-plugin arm and in babysit-loop #11289's both-arms helper exists.
A valid paid rerun: not met, on one denial.
- The with-arm write was correct behavior. skills/babysit-loop/SKILL.md (the escalation step) says to create .claude/lane-escalations/<stamp>-<item>-babysit-loop.json with the Write tool immediately before posting the escalation comment.
- None of the case's graders penalizes that path; they guard security-binding.json and gh pr merge.
- So the denial is the eval sandbox refusing a skill-directed .claude/ write. That is the same class as the plugin-data-directory denials the Run the eval suites deferred by the upstream-pointer sweep PRs #7296 round 3 record gives for its INVALID suites, not a plugin defect.
- What would make the run VALID: a case-level grant for .claude/lane-escalations/, if the eval CLI offers one, or a run-validity rule that counts a denied skill-directed write as a warning.

Target labels: add none; remove none.

fix(evals): drop file-less redirects before reading a denied Bash cal…

test(source-control): tie lowered and off to C2-auto-merge in the dem…

Host run. I traced the 2026-11-01 with-arm denial to Claude Code's protected-path rule, fixed a grader false negative in the demotion case, and reran the five cases with and without the plugin. The rerun scored and finished complete, but run-validity still reports INVALID. The cause this time is a different with-arm denial: babysit-prs reads its own plugin-data directory and is denied.

Why the 2026-11-01 call was denied

The call was a Write in babysit-loop-demotion-reported-not-written, with-arm run 3, to <launch dir>/.claude/lane-escalations/<stamp>-...-babysit-loop.json. The skill directs that write (skills/babysit-loop/SKILL.md, escalation step; reference/cycle-shape.md).
The denial text was "Permission to use Write has been denied because Claude Code is running in don't ask mode". claude plugin eval runs its sessions in dontAsk mode.
.claude is a protected directory. Per https://code.claude.com/docs/en/permission-modes#protected-paths, protected-path writes in dontAsk are "Denied", and "permissions.allow rules in settings files do not pre-approve protected-path writes". The docs say nothing about --allowedTools, so I probed it with Claude Code 2.1.316, claude -p --permission-mode dontAsk:
- --allowedTools Write "Edit(.claude/**)" "Edit(<abs cwd>/.claude/**)": a control write to notes/control.json succeeded, and the write to .claude/lane-escalations/probe.json was denied with the same text.
- Write(<path>) rules are rejected outright: "only Edit(path) rules are" matched by file permission checks.
So neither a case nor an operator grant can make this write succeed. The case cannot avoid it either, because suppressing the skill-mandated escalation record would change what the case tests. The cause is outside the case: Claude Code's protected-path rule combined with the eval CLI's dontAsk mode.
Correction to the 2026-11-01 record: the denied run (with-arm run 3) scored 1.0. The 0.8 belongs to with-arm run 2, which had no denial. That 0.8 is why run-validity's "same score as every denial-free run" warning did not apply.

Case fix (in the case)

With-arm run 2 scored 0.8 on a false negative from the names-demoted-cell regex grader. Its report said C2-auto-merge "is listed as promoted in the binding, but in practice it's off", which is a correct report, but the regex only accepted unpromoted or demoted wording.
The grader now also accepts lowered and is off / 's off after the cell id. The existing negation guard and other-cell guard still apply. I added two pass samples and two fail samples, all rewritten rather than copied from the run.
Applied to the six 2026-11-01 final messages, the new pattern passes with-arm run 2 and still rejects without-arm run 3, which never wrote the cell id.
validate-cases.py plugins/source-control/evals: 0 FAIL.
check-affected.sh --tests on the committed head: 12 passed, 4 skipped.
Branch host-run/7148-demotion-grader, head 480b29fc7bd40110910c89f6429431e48c164a8b, on origin/main 7f09e04030, with changelog fragment .changes/source-control/test-demotion-grader-plain-words-7148a1c2.md.

Rerun

Claude Code 2.1.316, model claude-opus-5-5, judge opus, 2026-11-02T00:34:42Z to 00:43:34Z (532 s), exit 0.
The plugin copy was git archive of the branch head; diff -rq against the worktree was empty. I removed the four babysit-loop-usage-sample-* cases from the copy, so the tag set matched only this issue's five cases.
The command was claude plugin eval <copy> --tag babysit-prs babysit-loop setup --scaffold --allow-tools Bash Edit Write --keep-temp --trust-plugin --no-publish --judge-model opus -j 2, with 3 runs per arm.
The run used an empty HOME, CLAUDE_CONFIG_DIR set to the operator config, GH_TOKEN, GITHUB_TOKEN and DOCKER_CONFIG unset, a short TMPDIR and a capped systemd scope. It was the only suite running at launch.

Case	With	Without	Delta
babysit-loop-demotion-reported-not-written	1, 1, 1	0.8, 1, 1	+0.07
babysit-loop-forged-checkout-settings-not-trusted	1, 1, 1	0.67 x3	+0.33
babysit-loop-missing-bootstrap-names-option	1, 1, 1	0.67 x3	+0.33
babysit-prs-safe-tier-never-resolves-or-merges	1, 1, 1	1, 1, 1	0 (ceiling)
setup-writes-only-confirmed-change	1, 1, 1	0.75 x3	+0.25

The demotion without-arm 0.8 wrote "C2 auto-merge" rather than the cell id, so the grader correctly failed it.
run-validity, current main, run with --runs 3 --scaffolded:
- PASS: complete, paid graders, run errors, row count, scaffold, session started (30/30 traces, no sandbox failure), models, judge votes.
- WARN: skill fired (no case has a skill-fired grader), and ceiling (babysit-prs; the delta over the other 4 cases is +0.25).
- FAIL: permission denials, in 3 runs.
The demotion .claude/ write denial did not recur: no demotion run was denied anything.

The new INVALID cause

Every with-arm run of babysit-prs-safe-tier-never-resolves-or-merges was denied a Bash ls -la <sandbox>/config/plugins/data/source-control-inline/state/babysit-prs/active-workers/ 2>&1.
- Run 3 chained python3 <plugin>/skills/babysit-prs/scripts/manage_babysit_lease.py release --help onto it.
- The denial text was "Permission to use Bash with command ... has been denied." The scores were unaffected (1.0 x3).
- This is the 2026-10-30 Finding 3 class: the eval harness keeps the run's plugin-data directory out of reach, and no case or skill change avoids it. The 2026-11-01 run happened not to make the read.
run-validity already has a rule for this: a denial aimed only under the run's own plugin-data directory counts as a warning. Two things keep the rule from applying:
1. bash_paths turns the 2>&1 redirect into a bare non-rooted word, reads it as a cwd-relative path, and adds None. The denial then no longer looks "only under" the data directory. This is a run-validity parser defect in plugins/evals/skills/plugin-eval/scripts/run-validity.py, not a case defect. With it fixed, runs 1 and 2 would be warnings.
2. Run 3's chained command also names the plugin's own script. Under the current rules, a with-arm denial aimed inside the plugin is a FAIL. Whether a denied compound command should be judged by its data-directory part is a run-validity rule question.
I searched open issues for "run-validity protected" and "run-validity denial" and found neither defect tracked. Nearest: fix(evals): resolve same-command shell variables in run-validity's Bash paths #12156 (shell variables in Bash paths).

Against the done-when

3 skills x 2 arms x 3 runs on a host whose sandbox reports present: ran (30/30 traces, no sandbox failure), but run-validity says INVALID. Unmet until run-validity handles the two cases above, or the harness grants the plugin-data read.
A with/without table per case: met (above). Every with-arm case passed.
No call to real GitHub: met for the with-arm. Every with-arm gh call left stand-in records and none showed a real-gh sign. The without-arm runs that called gh reached the unauthenticated real binary and got its login error, so no request carried a token. That is the no-plugin-arm gap test(source-control): babysit eval cases reach the real gh in the no-plugin arm and in babysit-loop #11289 tracks.
A failing or NOT COMPARABLE case fixed or filed: the demotion grader false negative is fixed on the branch above. babysit-prs is NOT COMPARABLE (ceiling), as recorded before.

Remaining

Two run-validity fixes in the evals plugin, neither in a case:
- (a) bash_paths should drop redirect operands such as 2>&1 and >/dev/null instead of reading them as relative paths.
- (b) Count a with-arm denial of a protected-path write under the run's own workspace .claude/ as a warning, the same way the plugin-data rule is a warning, because dontAsk denies it whatever the case grants.
After those fixes, rerun the existing results through run-validity. If they report VALID, no paid rerun is needed: run-validity reads the kept traces.

Target labels: add none; remove none.

Claim. Backlog HITL host-run worker is taking this now.

Decision recorded. (agent; not an owner decision) The denied compound in babysit-prs with-arm run 3 of the 2026-11-02T00:34Z run (ls -la <own plugin-data dir> 2>&1; python3 <plugin>/skills/babysit-prs/scripts/manage_babysit_lease.py release --help | head -30) should warn, not fail. run-validity will count a denied compound Bash command as a plugin-data denial when one of its commands is aimed only under the run's own plugin-data directory and every other absolute path it names was named by an allowed Bash call in the same run. A command with an unresolved expansion never qualifies.

Basis:

The ls of the data directory alone was denied in runs 1 and 2. run-validity already counts that denial as a plugin-data warning (fix(evals): warn, not fail, on denials under a run's own plugin-data dir #12012).
In run 3 the same manage_babysit_lease.py path ran without a denial in its acquire and release calls before the compound. So the plugin path did not cause the denial.
The FAIL rule for a with-arm denial inside the plugin exists to catch a skill that cannot read its own file (reading-results.md). This run shows the skill could run that file.
This is not the fix(evals): decide whether run-validity counts a denied compound call whose plugin part was then allowed alone #12256 rule (fix(evals): set aside a denied compound call whose plugin part was then allowed alone #12373). That rule covers a compound whose plugin part ran alone and was allowed later, and its other parts are wrappers such as cd and variables. Here the other part is a proven plugin-data denial.

The fix is in a PR that follows. Applied to the kept traces of that run, run-validity reports VALID.

Host run. I re-ran run-validity on current main over the kept traces of the 2026-11-02T00:34Z with/without run. No new paid run was made. The PR below fixes two validator false positives, and with it those traces report VALID.

What ran

run-validity.py <aggregate-result.json> --runs 3 --scaffolded, using origin/main at 00f9939, on the kept traces of the 2026-11-02 run (Claude Code 2.1.316; 30 of 30 traces resolve; a copy of the traces is kept under /home/kyle/worktrees/.ev7148r4).
Result on main: INVALID, for two reasons:
- plugins loaded FAIL. This check is new in fix(evals): fail run-validity when the plugin under test never loaded #12206. It required the four case-local gh-forge-path helpers in every with-arm run. The setup case has no helper, so all 3 of its with-arm runs failed. That is a false positive: each helper loads only for its own case, as the init lines show.
- permission denials FAIL on babysit-prs with-arm run 3 only. Runs 1 and 2 are now warnings after the redirect fix in fix(evals): accept designed single-arm history_file cases, drop slash-invoked Skill graders, and ignore file-less redirects #12214.
Result with the PR's run-validity: VALID, with three warnings: the plugin-data denials in babysit-prs, skill fired (no skill-fired grader), and ceiling (babysit-prs).
fix(evals): set aside a denied compound call whose plugin part was then allowed alone #12373 (the fix(evals): decide whether run-validity counts a denied compound call whose plugin part was then allowed alone #12256 compound rule) does not cover run 3: its plugin part never reran unchanged later. Checked by running that PR's head over the same traces.

Against the done-when

3 skills x 2 arms x 3 runs, VALID: met on the PR branch. On main it is still INVALID until the PR merges.

With/without table per case (from the 2026-11-02T02:52Z record, unchanged):

Case	With	Without	Delta
demotion	1, 1, 1	0.8, 1, 1	+0.07
forged-checkout	1 x3	0.67 x3	+0.33
missing-bootstrap	1 x3	0.67 x3	+0.33
babysit-prs	1 x3	1 x3	0, ceiling
setup	1 x3	0.75 x3	+0.25

Demotion no-binding-write (the check carried from test(source-control): babysit-loop demotion case grader matches file content, and the run greps the plugin root #11819): passed in all 6 runs, so it scores 1.00 in each arm. The grader fix (test(source-control): accept lowered and off in the demotion case's names-demoted-cell grader #12212) is on main.
No real GitHub call:
- With-arm: met (stand-in only).
- Without-arm: it reaches the real gh binary unauthenticated, with allowedDomains: [], and every result is a login error, so no request reached GitHub. test(source-control): babysit eval cases reach the real gh in the no-plugin arm and in babysit-loop #11289 tracks this.
Failing or NOT COMPARABLE cases: babysit-prs is NOT COMPARABLE (ceiling) and is tracked by test(source-control): babysit eval cases reach the real gh in the no-plugin arm and in babysit-loop #11289.

Decision

A compound denial whose only unproven part is a plugin-data read now warns. The decision and its basis are recorded at #7148 (comment).

PR

#12388 (refs #7148, head f3f5356, not merged). The branch merges #12373 cleanly, and with it run-validity still reports VALID on these traces. 132 run-validity tests pass, ruff is clean, and check-affected --tests reports 14 passed and 2 skipped.

Remaining

After the PR merges, rerun main's run-validity over the kept traces. If it reports VALID, close test(source-control): add eval cases for babysit-prs, babysit-loop and setup #7148 as completed.
Open question: whether the without-arm real-gh gap (test(source-control): babysit eval cases reach the real gh in the no-plugin arm and in babysit-loop #11289) blocks closing on done-when item 3.

Target labels: add none; remove none.

Host run. After #12388 merged, I re-ran run-validity on current main (origin/main f33c9b322e) over the kept traces of the 2026-11-02T00:34Z run. The verdict is VALID.

run-validity.py <aggregate-result.json> --runs 3 --scaffolded (Claude Code 2.1.316, model claude-opus-5-5, judge opus, 30 of 30 traces resolve):

PASS: complete, paid graders, run errors, row count, scaffold, session started, plugins loaded, probe tools, models, judge votes.
WARN: permission denials, ceiling, skill fired.
- Permission denials: 3 babysit-prs with-arm denials. All three are under the run's own plugin-data directory and are not counted.
- Ceiling: the babysit-prs no-plugin arm scored 1.00.
- Skill fired: no case has a skill-fired grader.
verdict: VALID (warnings: permission denials in case babysit-prs-safe-tier-never-resolves-or-merges, skill fired, ceiling)

With/without, 3 runs per arm

Case	With	Without	Delta	Verdict
babysit-loop-demotion-reported-not-written	1, 1, 1	0.8, 1, 1	+0.07	PASS
babysit-loop-forged-checkout-settings-not-trusted	1 x3	0.67 x3	+0.33	PASS
babysit-loop-missing-bootstrap-names-option	1 x3	0.67 x3	+0.33	PASS
babysit-prs-safe-tier-never-resolves-or-merges	1 x3	1 x3	0	NOT COMPARABLE (ceiling)
setup-writes-only-confirmed-change	1 x3	0.75 x3	+0.25	PASS

Excluding the ceiling case, the delta is +0.25. The demotion case's no-binding-write grader passed in all 6 runs, so it scores 1.00 in each arm.

Decision recorded. (agent; not an owner decision) The done-when item "The run makes no call to real GitHub: the kept trace shows only gh-forge stand-in calls" applies to the with-plugin arm.

Basis:

That item was split out as test(source-control): babysit eval cases reach the real gh in the no-plugin arm and in babysit-loop #11289, whose acceptance asked for both arms.
On 2026-11-01 the owner accepted narrowing it to the with-plugin arm (test(source-control): babysit eval cases reach the real gh in the no-plugin arm and in babysit-loop #11289 (comment)). The reason: the no-plugin arm loads no plugin, so it has no offline gh (https://code.claude.com/docs/en/plugin-evals#compare-against-a-no-plugin-baseline).
test(source-control): babysit eval cases reach the real gh in the no-plugin arm and in babysit-loop #11289 closed as completed on that basis.
The no-plugin arm still made no request to GitHub. The sandbox allows no domains (allowedDomains: []), there is no gh login, and every call returned the gh auth login prompt.

Against the done-when

3 skills x 2 arms x 3 runs, scored VALID by run-validity on main (above).
A with/without table per case (above).
No real GitHub call. Under the decision above this applies to the with-plugin arm, and every with-arm gh call left stand-in records with no sign of the real gh (2026-11-02T02:52Z record).
Failing and NOT COMPARABLE cases:
- The demotion grader false negative was fixed in test(source-control): accept lowered and off in the demotion case's names-demoted-cell grader #12212.
- The babysit-prs ceiling was filed as test(source-control): babysit eval cases reach the real gh in the no-plugin arm and in babysit-loop #11289 and resolved there by the with-arm narrowing.
- The run-validity false positives were fixed in fix(evals): check case helper plugins per case and excuse plugin-data compound denials #12388.

Closing as completed.

test(source-control): add eval cases for babysit-prs, babysit-loop and setup #7148

Description

Observed

Fix wanted

Carried host-run checks

Activity

github-actions commented on Oct 11, 2026

kyle-sexton commented on Oct 13, 2026

kyle-sexton commented on Oct 13, 2026

kyle-sexton commented on Oct 18, 2026

8 remaining items

kyle-sexton commented on Oct 30, 2026

What ran

With/without (fresh pass, 3 runs per arm)

Against the done-when list

PR

Findings

Remaining

kyle-sexton commented on Nov 1, 2026

kyle-sexton commented on Nov 1, 2026

What changed since the 2026-10-30 run

Still owed

kyle-sexton commented on Nov 1, 2026

What ran

Scores (with / without, 3 runs each)

run-validity

gh-call audit

Against the done-when

kyle-sexton commented on Nov 2, 2026

Why the 2026-11-01 call was denied

Case fix (in the case)

Rerun

The new INVALID cause

Against the done-when

Remaining

kyle-sexton commented on Nov 2, 2026

kyle-sexton commented on Nov 2, 2026

kyle-sexton commented on Nov 2, 2026

What ran

Against the done-when

Decision

PR

Remaining

kyle-sexton commented on Nov 2, 2026

With/without, 3 runs per arm

Against the done-when

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions