[kyle-sexton]

No related issue: owner-approved follow-up to #10077 (session 2026-10-25, Q2 option a)

Summary

The session started check (#10077) found a sandbox that never started only by matching one Claude Code error string, so a reworded message would let no-shell runs pass as VALID again. This adds a second signal that does not depend on any wording: a run with 2 or more Bash calls, every one failed.

Fix

• run-validity.py: sandbox_failed becomes shell_failure, which returns a reason. It returns the sandbox-start text when present, else "all N Bash calls failed" when a run made at least MIN_BASH_CALLS (2) Bash calls and every one came back is_error. A lone failed call (such as a grep that matches nothing) does not count.
• Two tests: an all-failed run with new wording is INVALID, and a single failed call or a mixed run passes.

Verification

• run-validity.test.sh: 59 tests OK. The new INVALID test failed before the change.
• Across 206 kept eval traces on this host: 0 of 14 healthy short-TMPDIR runs had every Bash call fail, while 97 of 123 default-/tmp runs did. Their commands run but report Exit code 1 under the read-only temp directory ([BUG] plugin eval: sandbox denies every write in the agent's workspace — //tmp denyWrite covers the /tmp/claude-eval-* root anthropics/claude-code#104631), so the check also catches that broken class, which the gate passed before.
• The real probe results are unchanged: the default-TMPDIR run stays INVALID, and the short-TMPDIR run and the feat(harness-ops): show per-role lane spend in the morning brief #9494 rerun stay VALID.
• Left out: a healthy suite whose cases deliberately make every Bash call fail would now be flagged. I know of none, and the 14 healthy traces had none.

Related

• Refs fix(evals): fail eval runs that never started or had no sandbox #10077, evals: run-validity calls runs VALID when every Bash call failed to start the sandbox (long TMPDIR) #9611

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-25T19:26:12.040488Z	c47e8ad	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[github-actions]

claude-security-review reviewed 8bba0271fc59261f1af8a759083a8b3728ebea60: 0 finding(s), 0 as inline comments.

Incremental review of run-validity.py and test_run_validity.py since c47e8ad. I found no security issues.

The change adds is_denial(), which shell_failure now uses to drop permission-denied Bash results from its failure count. denials() reuses the same helper. Neither path adds an injection surface, a privilege boundary or any handling of secrets. The script only parses local trace JSONL. It uses json.dumps and a substring check, and it runs no command and does no path or template handling.

The behavior change is that denied Bash calls no longer trigger the "every Bash call failed" invalid-run signal. They are still judged by check_denials, and the new test checks that the permission denials check does not report PASS. That check is not weakened, so a run cannot pass validity by hiding denials.

No trusted-path files changed in this increment, and there are no manifest changes. The PR context contained no embedded directives.

critical-findings: 0

[github-actions]

claude-review: 1 inline finding(s) on c47e8ad961566442b406028afdd3d1224a07b8f3; the summary is in this lane's pull request comment.

claude-lane: claude-review

[github-actions]

claude-review reviewed 8bba0271fc59261f1af8a759083a8b3728ebea60: 0 finding(s), 0 as inline comments.

Incremental review of 8bba027 against c47e8ad: no findings.

The change extracts is_denial and uses it in two places, shell_failure and denials. In shell_failure, a denied Bash call is dropped from bash_ids and skipped, so a permission-denied call no longer counts as a failed shell call. check_denials judges it instead.

• is_denial refactor (run-validity.py:345-349): its condition is the same as the inline one it replaces in denials (is_error true and DENIAL_TEXT in the content). I found no change to denials behavior.
• shell_failure discard (run-validity.py:367-369): it only runs for results whose id is in bash_ids, and the tool_use block precedes its result. Discarding before the sandbox-text check is correct, because a denial never reached the shell. Mixed cases work as intended: one denial plus one real failure leaves a single counted call, which is under MIN_BASH_CALLS. One denial plus two real failures reports "all 2 Bash calls failed".
• Test (test_run_validity.py:353-358): two denied Bash calls would have been INVALID ("all 2 Bash calls failed") before this change. Now session started passes and permission denials is evaluated and does not pass. The test would fail against the previous behavior, so it covers the change. I read the code and test statically and did not run the suite.

Spend: none beyond ordinary lane review runs.