fix(evals): count sandbox network denials in run-validity

.changes/evals/fix-12988-run-validity-network-denials-3b7f2e91.md

@@ -0,0 +1,7 @@
+---
+bump: patch
+---
+
+### Fixed
+
+- `/evals:plugin-eval` run-validity counts a with-arm Bash result's sandbox network denial as a permission denial and names the denied hosts in the summary: it is a FAIL, and a warning in an invocation-probe or control case or one tagged `no-network`; a without-arm run's network denials are not read

plugins/evals/skills/plugin-eval/SKILL.md

@@ -296,10 +296,14 @@ Read in this order. Stopping early at any step is the finding.
 
    A with-arm denial aimed at or under the plugin's directory, at a directory above it (which covers
    its files), or with no absolute path or no known plugin directory, is a FAIL: the agent reached
-   for a plugin file it could not read, so the fact belongs in the hub. The exceptions that turn a
+   for a plugin file it could not read, so the fact belongs in the hub. So is a with-arm Bash call
+   whose result reports a sandbox network denial (Sources: sandbox network denial report), whose
+   hosts the summary names: the score measured a blocked request. Only an invocation probe, a
+   control, or a case whose `prompt.md` tags include `no-network` turns it into a warning; a
+   without-arm one is not read. The exceptions that turn a
    denial into a warning (a denial the environment makes in both arms, such as a user-scope read of
    `~/.claude/standards`; a denial in a case whose denial-free runs scored the same; an invocation
-   probe's denial; a negative control's broken side, run with `--own-copy-search-warns`, searching its own copy, where such a case cannot decide the control), Bash denial paths, A/B gating,
+   probe's denial; a sandbox network denial in an invocation probe, control or `no-network` case; a negative control's broken side, run with `--own-copy-search-warns`, searching its own copy, where such a case cannot decide the control), Bash denial paths, A/B gating,
    the 0-turn `context: fork` run the gate passes, and invocation-probe misses and tools: [reading-results.md, Denials](reference/reading-results.md#denials).
 2. `partial`. `true` (with `partialReason` of `cost_ceiling`, `interrupted`, or `auth_failed`) means
    the suite did not finish: report that and keep the document out of any trend.
@@ -465,6 +469,7 @@ record a drift outcome in this plugin's CHANGELOG.
 | Sandbox backend table and the no-backend refusal | [Set up Linux and WSL2](https://code.claude.com/docs/en/sandboxing#set-up-linux-and-wsl2); plugin-evals platform notes | 2026-10-04 | Either names a Windows backend, or adds or drops a package or WSL note |
 | Docker credential-store refusal and the `~/.docker/bin` exemption | <https://code.claude.com/docs/en/changelog> (2.1.293); no docs page covers it | 2026-10-10 | A docs page documents the check, or a release note touches sandbox credential exclusion or exempt paths |
 | `eval_tmpdir` 24-character bound and its refusal | [unix(7)](https://man7.org/linux/man-pages/man7/unix.7.html) `sun_path`; sandbox-runtime's bridge sockets and the 63- and 24-character pair in [sandboxes.md, Sandbox start](reference/sandboxes.md#sandbox-start) | 2026-10-31 | sandbox-runtime renames its bridge sockets, a release note touches `plugin eval` sandbox paths or `TMPDIR`, or a `<dir>` within the bound fails with the bridge-socket error |
+| Sandbox network denial report, read by the permission denials check | [Sandboxing, The unsandboxed retry escape hatch](https://code.claude.com/docs/en/sandboxing#the-unsandboxed-retry-escape-hatch) says only that the result names the denied host; no page documents the report's format, observed in a with-arm Bash result on Claude Code 2.1.317 (issue #12988), its shape kept in `scripts/fixtures/run-validity/traces/network-denied.jsonl` | 2026-11-04 | That page documents the violation report, a release note touches sandbox violation reporting, or a kept trace shows a network denial the check missed |
 | Cloud route host and packages | [Installed tools](https://code.claude.com/docs/en/cloud-environments#installed-tools) | 2026-10-11 | That section changes its host description or package list |
 | Read-only tool set | <https://code.claude.com/docs/en/plugin-evals#grant-tools> | 2026-10-10 | That section's allowed-tools sentence changes |
 | Worktree Bash guard refusal of `claude plugin eval --help`, and which session shapes it covers | Observed on Claude Code 2.1.316 ([records.md](reference/records.md#worktrees)) | 2026-10-31 | A release note touches worktree isolation or `plugin eval` |

plugins/evals/skills/plugin-eval/reference/reading-results.md

@@ -102,6 +102,14 @@ above the plugin's directory, or with no absolute path or no known plugin direct
   A/B pass, gate each version's result file.
 - Any other denial, in either arm, is a warning only when its case has denial-free runs in that arm
   and its run scored the same as each; otherwise it is a FAIL.
+- A with-arm Bash call whose result reports a sandbox network denial, error or not, is a denial
+  whose hosts the summary names (the report and its source:
+  [SKILL.md, Sources](../SKILL.md#sources), sandbox network denial report). It is a FAIL, since the
+  score then measures a blocked request, unless the case is an invocation probe, a control (tagged
+  `control`, `no-trigger` or `negative-control`, described as a no-trigger case, or forbidding its
+  own skill's call and requiring none), or tagged `no-network`; those set it aside as a warning.
+  A without-arm run's network denials are not read, and the report quoted anywhere but a Bash
+  result is not a denial.
 - A run at 0 turns whose trace holds assistant messages ran a slash-invoked `context: fork` skill
   in a forked session, and the gate lists it under `session started` without failing it. Any run
   fails there whose trace holds a user message with a `local-command-stderr` reading `Shell command
@@ -130,7 +138,7 @@ above the plugin's directory, or with no absolute path or no known plugin direct
 - An invocation probe is a case whose `prompt.md` tags include `invocation-probe`, as
   `/skill-quality:check measure-invocation` emits it. Its should-fire misses are the trigger rate
   it measures: `skill fired` reports them as a warning with per-case counts, never a FAIL. Run
-  errors and denials keep their full force, apart from the two probe exceptions above and a
+  errors and denials keep their full force, apart from the probe exceptions above and a
   max-turns error after the graded Skill call. `probe tools` warns when a probe run could call
   Read, Glob, Grep or Agent, the tools the emitted README's run line removes, or its trace has no
   init tools list; when no probe run's list can be read, the check is UNCHECKED.

plugins/evals/skills/plugin-eval/scripts/fixtures/run-validity/traces/network-denied.jsonl

@@ -0,0 +1,4 @@
+{"type":"system","subtype":"init","model":"claude-opus-5-5","tools":["Skill"]}
+{"type":"assistant","message":{"content":[{"type":"tool_use","id":"n1","name":"Bash","input":{"command":"curl -sS -o /dev/null -w '%{http_code} exit=%{exitcode}' https://xtomd.com/post"}}]}}
+{"type":"user","message":{"content":[{"tool_use_id":"n1","type":"tool_result","content":"curl: (7) CONNECT tunnel failed, response 403\n000 exit=7\n<sandbox_violations>\ndeny network-outbound xtomd.com:443 (user denied)\n</sandbox_violations>","is_error":false}]},"tool_use_result":{"stdout":"curl: (7) CONNECT tunnel failed, response 403\n000 exit=7","stderr":"<sandbox_violations>\ndeny network-outbound xtomd.com:443 (user denied)\n</sandbox_violations>"}}
+{"type":"result","subtype":"success","is_error":false,"permission_denials":[]}

plugins/evals/skills/plugin-eval/scripts/run-validity.py

@@ -123,7 +123,12 @@
                       the graded Skill call, or the run is a should-not-fire
                       run that held while every denial-free run of its case
                       held; a should-fire miss with a denial takes the rules
-                      above
+                      above. In a with-arm run, a Bash tool result whose
+                      sandbox report denies an outbound host (SANDBOX_VIOLATIONS)
+                      is a denial too, its hosts named in the summary: in an
+                      invocation-probe, control or no-network-tagged case it
+                      is set aside as a warning, otherwise a FAIL. A
+                      without-arm run's network denials are not read
     skill fired       every with-arm run of a should-trigger case fired the skill;
                       in an invocation-probe case, or one named by
                       --measured-fire, a miss is the trigger rate it measures,
@@ -252,6 +257,13 @@
 TOOL_USE, USE_ID, USE_NAME, USE_INPUT = "tool_use", "id", "name", "input"
 TOOL_RESULT, RESULT_USE_ID, RESULT_ERROR = "tool_result", "tool_use_id", "is_error"
 DENIAL_TEXT = "denied by your permission settings"  # in an is_error tool_result
+# A Bash tool_result's sandbox report, is_error or not, and its network denial
+# lines. Upstream documents no format: observed in a with-arm trace of the x
+# plugin's suite on Claude Code 2.1.317, 2026-11-03; network-denied.jsonl in the
+# test fixtures keeps its shape. Pointer and recheck trigger: SKILL.md, Sources,
+# "Sandbox network denial report".
+SANDBOX_VIOLATIONS = re.compile(r"<sandbox_violations>(.*?)</sandbox_violations>", re.S)
+NETWORK_DENIAL = re.compile(r"^deny network-outbound (\S+)", re.M)
 # Anywhere in a trace line when the shell sandbox never started (Claude Code 2.1.310).
 SANDBOX_FAILURE_TEXT = "Sandbox is required but failed to initialize"
 # A user line's local-command-stderr when a slash-invoked skill's pre-computed
@@ -355,6 +367,8 @@
 # emit-plugin-eval): a miss is the rate it measures, and a run that hit its turn
 # cap after the Skill call was already graded.
 PROBE_TAG = "invocation-probe"
+# A case that declares it needs no network: a sandbox network denial there warns.
+NO_NETWORK_TAG = "no-network"
 # The whole error text, optionally after the runner's "exit N: " prefix.
 MAX_TURNS_ERROR = re.compile(
     r"(?:exit \d+: )?Reached maximum number of turns \(\d+\)\.?"
@@ -551,6 +565,10 @@ def is_probe(case, root):
     return PROBE_TAG in prompt_fields(case, root).get("tags", [])
 
 
+def is_no_network(case, root):
+    return NO_NETWORK_TAG in prompt_fields(case, root).get("tags", [])
+
+
 def is_control(case, root):
     """Control: tagged or described as one, else it forbids own-skill calls and
     requires none (a should-trigger case may also carry a max-0 guard)."""
@@ -1589,6 +1607,45 @@ def denials(run):
     return out
 
 
+def result_text(content):
+    """A tool_result's text: its string content, else its text blocks joined."""
+    if isinstance(content, str):
+        return content
+    return "\n".join(
+        b if isinstance(b, str) else str(as_dict(b).get("text") or "")
+        for b in content if isinstance(b, (str, dict))
+    ) if isinstance(content, list) else ""
+
+
+def network_denials(run, skip=()):
+    """In a with-arm run, every Bash call, its id not in skip, whose tool_result
+    reports a sandbox network denial (SANDBOX_VIOLATIONS, NETWORK_DENIAL), shaped
+    as denials gives them: no path, so no rule reads it as aimed outside the
+    plugin, and the denied hosts as its target. Text quoting the report anywhere
+    but a Bash tool_result is not a denial; a without-arm run has none."""
+    if run.arm != "with":
+        return []
+    bash, out = set(), []
+    for line in run.trace:
+        for block in as_list(as_dict(line.get(MESSAGE)).get(CONTENT)):
+            use_id = block.get(RESULT_USE_ID)
+            if block.get(BLOCK_TYPE) == TOOL_USE and block.get(USE_NAME) == BASH_TOOL:
+                bash.add(block.get(USE_ID))
+            elif block.get(BLOCK_TYPE) == TOOL_RESULT and use_id in bash and use_id not in skip:
+                hosts = tuple(dict.fromkeys(
+                    host
+                    for report in SANDBOX_VIOLATIONS.findall(result_text(block.get(CONTENT)))
+                    for host in NETWORK_DENIAL.findall(report)
+                ))
+                if hosts:
+                    out.append((
+                        "%s: Bash network-outbound %s (%s)"
+                        % (run.where, ", ".join(hosts), run.trace_path),
+                        (None,), BASH_TOOL, use_id, ("network-outbound", hosts), None,
+                    ))
+    return out
+
+
 def graded_call_at(run, root):
     """(tool_use_id -> position of each tool_use in the trace, position of the first
     Skill call a Skill grader's input_match accepts, or None)."""
@@ -2090,6 +2147,15 @@ def check_denials(runs, roots, root, own_copy_search_warns=False):
         ]
         for r in traced
     }
+    # With-arm sandbox network denials: set aside in an invocation-probe, control
+    # or no-network case, else a FAIL whatever the rules below say.
+    network = {r: network_denials(r, {d[3] for d in every[r]}) for r in traced}
+    offline = {
+        r: network[r]
+        if is_probe(r.case, root) or is_control(r.case, root) or is_no_network(r.case, root)
+        else []
+        for r in traced
+    }
     found = {
         r: [
             d
@@ -2099,20 +2165,35 @@ def check_denials(runs, roots, root, own_copy_search_warns=False):
             and d not in searched[r]
             and d not in recovered[r]
         ]
+        + [d for d in network[r] if d not in offline[r]]
         for r in traced
     }
-    denied = [r for r in traced if every[r]]
+    denied = [r for r in traced if every[r] or network[r]]
     warned = [
         r
         for r in denied
-        if not found[r]
-        or probe_denial(r, found, runs, root)
-        or same_as_clean(r, found, runs, roots)
-        or in_every_run(r, found, runs, roots)
+        if len(network[r]) == len(offline[r])
+        and (
+            not found[r]
+            or probe_denial(r, found, runs, root)
+            or same_as_clean(r, found, runs, roots)
+            or in_every_run(r, found, runs, roots)
+        )
     ]
     failed = [r for r in denied if r not in warned]
-    details = [line for r in denied for line, _, _, _, _, _ in every[r]]
+    details = [line for r in denied for line, _, _, _, _, _ in every[r] + network[r]]
     summary = "%s in %s" % (plural(len(details), "denial"), plural(len(denied), "run"))
+    blocked = sum(len(network[r]) for r in denied)
+    if blocked:
+        summary += "; %s, outbound to %s" % (
+            plural(blocked, "sandbox network denial"),
+            ", ".join(sorted({h for r in denied for d in network[r] for h in d[4][1]})),
+        )
+    unplugged = sum(len(offline[r]) for r in denied)
+    if unplugged:
+        summary += "; %s in an invocation-probe, control or no-network case, not counted" % plural(
+            unplugged, "sandbox network denial"
+        )
     set_aside = sum(len(data[r]) for r in denied)
     if set_aside:
         summary += "; %s under the run's own plugin-data directory, not counted" % plural(
@@ -2140,7 +2221,8 @@ def check_denials(runs, roots, root, own_copy_search_warns=False):
     for r in denied:
         if is_probe(r.case, root) and r.case not in probe_cases:
             probe_cases.append(r.case)
-    details += [probe_split(c, runs, found, root) for c in probe_cases]
+    split = {r: found[r] + offline[r] for r in traced}
+    details += [probe_split(c, runs, split, root) for c in probe_cases]
     if warned:
         summary += (
             "; warnings (an invocation probe denied only calls other than Skill, after"
@@ -2151,7 +2233,7 @@ def check_denials(runs, roots, root, own_copy_search_warns=False):
             " run of its case was denied in both arms, none inside the plugin; and"
             " any named call other than Skill aimed only under the run's own"
             " plugin-data directory, or a Read, Grep, Glob or read-only Bash pipeline"
-            " aimed only at the session's own tool-results directory,%s%s which never"
+            " aimed only at the session's own tool-results directory,%s%s%s which never"
             " counts toward a"
             " FAIL; a run's score was taken under such a denial, so compare it only"
             " with runs denied the same way): %s"
@@ -2161,6 +2243,10 @@ def check_denials(runs, roots, root, own_copy_search_warns=False):
                 " its case names," if own_copy_search_warns else "",
                 " or a with-arm denied compound Bash call whose plugin part a later"
                 " call in its run ran alone and was allowed," if rerun else "",
+                " or a with-arm sandbox network denial in an invocation-probe, control"
+                " or no-network case,"
+                if unplugged
+                else "",
                 ", ".join(warned_cases),
             )
         )