fix(evals): run-validity passes runs whose fetch the sandbox's network block denied

This was generated by AI.

Problem

run-validity.py passes a run whose traces carry a sandbox network denial the case depends on. In the #12882 x suite smoke, 17 with-arm traces each hold deny network-outbound xtomd.com:443 inside a <sandbox_violations> block of a Bash tool result, and curl exits 7. run-validity.py (origin/main) reported check permission denials: PASS (no denied tool call in 37 traces). The cases those runs belong to all depend on the fetch, so their scores measure a blocked request rather than the skill.

This is the same class as the own-data write denial (own_data_denial): the sandbox, not the plugin, decided the outcome.

Fix

Count a deny network-outbound sandbox violation in a with-arm trace as a denial in the permission-denials check (or a new network check), and report the host. A case that declares it needs no network, or a negative control, can opt out the way invocation probes do.

Refs: #12882

Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.

Size: Small
Complexity: Low
Uncertainty: Known
Flags: none
Blocked by: none

Understood: run-validity.py passes with-arm runs whose Bash fetch was denied by the sandbox (deny network-outbound <host>), so the scores measure a blocked request. The fix is to count those violations in the permission-denials check and name the host.

Decision recorded. Collect each with-arm deny network-outbound sandbox violation as a new denial source in the permission-denials check, fail on it, name the host, and let invocation probes, controls and cases tagged no-network warn instead.
Agent choice, not an owner answer.
Written by the Issue Triage session from the issue body, intake's refinement, its events, and origin/main at 1a588be.
[quote truncated; the full decision is at the link]

#12988 (comment)

This decision settles the open questions (fold into the existing check, opt-outs, FAIL vs WARN, scope). Nothing further is needed from the reporter. No duplicates found among open issues.

_{Automated triage, applied by the lanes App.}

Not admitted to the issue lane yet: no trusted-human acceptance criteria: a listed human must put the confirmed - [ ] criteria in the body or post them as their own comment, or, on an issue a listed human filed, intake's newest refinement comment must hold them unedited; intake's refinement of this body is due and may supply them.

Refinement. A read-only run explored this repository, searched its issues and pull requests, and read the allowed docs to draft what follows. On an issue a listed human filed, it counts as the issue's spec once posted, while unedited; it is still model text, so check it.

Intended outcome: run-validity.py counts a with-arm deny network-outbound <host> sandbox violation as a denial, names the host, and lets no-network cases and negative controls opt out.

Related work

fix(evals): judge invocation-probe runs by their measured rate in run-validity #11639 (related): Added the invocation-probe handling in run-validity that the issue cites as the opt-out model.
fix(evals): warn on non-Skill denials in invocation-probe runs #11655 (related): Warns on non-Skill denials in invocation-probe runs; same denial check.
fix(evals): eval runs cannot read the plugin copy's own files, so skills that use them measure low #12880 (related): Another eval-run sandbox limit that distorts scores; same class.

Open questions

Should a network denial be folded into the existing permission denials check or reported as a new network check?
RECOMMENDED: Fold it into check_denials as a new denial source so the existing FAIL/WARN logic and summary apply, and name the host in the summary. Basis: .base/plugins/evals/skills/plugin-eval/scripts/run-validity.py:2058
How does a case declare it needs no network? No such field exists; the only opt-outs are the invocation-probe tag and CONTROL_TAGS in prompt frontmatter.
RECOMMENDED: Reuse is_probe and is_control as the opt-outs and add no new tag; add an explicit no-network tag only if the owner wants one. Basis: .base/plugins/evals/skills/plugin-eval/scripts/run-validity.py:349
Should a network denial in a with-arm run give FAIL or WARN?
RECOMMENDED: FAIL, because the score then measures a blocked request; probe and control cases warn. Basis: .base/plugins/evals/skills/plugin-eval/scripts/run-validity.py:2113

Sources read

.base/plugins/evals/skills/plugin-eval/scripts/run-validity.py:2058: check_denials builds the denial sets (own data, own results, own copy, recovered) and decides FAIL or WARN; no network handling.
.base/plugins/evals/skills/plugin-eval/scripts/run-validity.py:349: CONTROL_TAGS and PROBE_TAG are the existing opt-out markers.
.base/plugins/evals/skills/plugin-eval/scripts/run-validity.py:550: is_probe and is_control read the case prompt frontmatter tags.

To answer, post a comment that starts with this line:

**Decision recorded.** <answer>

_{Automated refinement by the lanes App. Answer with a comment whose first line is **Decision recorded.**.}

Claimed by host-run lane 4 (melo-lap-001, session f674da34-2c6a-489c-94cf-8a8887b5023d), 2026-11-04T00:2xZ.

The refinement's three questions are implementation choices inside this issue's Fix section, so lane 4 takes the recommended answers:

Fold a with-arm deny network-outbound <host> sandbox violation into check_denials as a new denial source, naming the host in the summary (Basis: run-validity.py:2058).
No new tag: invocation probes and CONTROL_TAGS cases are the opt-outs (Basis: run-validity.py:349, :550).
FAIL for a with-arm network denial; probe and control cases WARN (Basis: run-validity.py:2113, the same split the own-data denial uses).

Posted by the lane 4 session. This was generated by AI.

Decision recorded. Collect each with-arm deny network-outbound sandbox violation as a new denial source in the permission-denials check, fail on it, name the host, and let invocation probes, controls and cases tagged no-network warn instead.
Agent choice, not an owner answer.
Written by the Issue Triage session from the issue body, intake's refinement, its events, and origin/main at 1a588be.

Fold in or a new check: fold in, as the refinement recommends. check_denials (plugins/evals/skills/plugin-eval/scripts/run-validity.py:2058) builds its summary and FAIL/WARN split from denials(run) (:1541), which collects only denied tool calls. A sandbox violation is text inside a Bash tool result whose call ran, so it needs its own collector feeding check_denials.
Opt-out: the refinement's answer drops scope. The body says a case that declares it needs no network, or a negative control, opts out. Controls reuse CONTROL_TAGS and is_control (:349, :554); invocation probes reuse PROBE_TAG and is_probe (:357, :550); the no-network declaration is a new prompt-frontmatter tag, no-network, read the same way.
FAIL or WARN: FAIL for a with-arm case outside those opt-outs, since its score measures a blocked request. The refinement's "controls warn" is not true of today's code: is_control is not among the warn rules in check_denials, so the control opt-out is new code, not reuse of a warn path.
Scope: with-arm traces only, as the body says. A fixture trace exercises it in the existing test suite; no live eval re-run is part of this issue.

run-validity.py collects each deny network-outbound <host> line in a <sandbox_violations> block of a Bash tool result in a with-arm trace as a denial, and the permission-denials check counts it.
The check's summary names each denied host.
A with-arm network denial fails the check, unless the case is an invocation probe, a control, or carries the no-network tag, which warn instead.
A fixture trace under plugins/evals/skills/plugin-eval/scripts/fixtures/run-validity/traces/ holds a deny network-outbound xtomd.com:443 violation, and test_run_validity.py asserts FAIL for it and WARN for a control and for a no-network case, and passes.

No description provided.

fix(evals): meet the #12988 checklist for network denials

test(x): tag the scaffolded read cases no-network

Navigation Menu

Repository navigation

fix(evals): run-validity passes runs whose fetch the sandbox's network block denied #12988

Problem

Fix

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

fix(evals): run-validity passes runs whose fetch the sandbox's network block denied #12988

Description

Problem

Fix

Activity

github-actions commented on Nov 3, 2026

melodic-automation-lanes commented on Nov 3, 2026

melodic-automation-lanes commented on Nov 3, 2026

kyle-sexton commented on Nov 4, 2026

kyle-sexton commented on Nov 4, 2026

github-actions commented on Nov 4, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions