From d45b25d500a3cb9f1a86fc3da4f734b0309dfdfa Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sun, 11 Oct 2026 11:33:22 -0400 Subject: [PATCH 1/3] docs(testing): record plugin-eval browser findings in the rubric and ADR 0056 ADR 0056 gains a dated Findings section from the eval-browser-dispatch probes on Claude Code 2.1.296: the eval hides a case's own files, narrow Bash grants match as written, playwright-cli cannot run inside plugin-eval Bash on Linux because the sandbox blocks Unix sockets, and CLAUDE_CODE_SUBPROCESS_ENV_SCRUB=1 breaks every eval Bash call. The run-e2e render checks say no catch rate is measured yet, point at the MCP-driven harness (#7137), and require a quoted rate to name its path. The rubric rows and the Claude in Chrome from WSL item are unchanged. Co-Authored-By: Claude Opus 5.5 --- .../docs-browser-rubric-p7-5c1e9a37.md | 7 ++++ ...nd-attach-produced-demo-evidence-to-prs.md | 35 +++++++++++++++++-- plugins/testing/skills/run-e2e/context/e2e.md | 12 +++++++ 3 files changed, 52 insertions(+), 2 deletions(-) create mode 100644 .changes/testing/docs-browser-rubric-p7-5c1e9a37.md diff --git a/.changes/testing/docs-browser-rubric-p7-5c1e9a37.md b/.changes/testing/docs-browser-rubric-p7-5c1e9a37.md new file mode 100644 index 0000000000..567d1a7655 --- /dev/null +++ b/.changes/testing/docs-browser-rubric-p7-5c1e9a37.md @@ -0,0 +1,7 @@ +--- +bump: patch +--- + +### Changed + +- `/testing:run-e2e`'s render checks say no per-check catch rate is measured yet: `playwright-cli` cannot run inside `claude plugin eval`'s Bash on Linux, so browser eval cases move to an MCP-driven harness (#7137), and a quoted catch rate names the path it measured, CLI or MCP. The browser-tool rubric rows are unchanged. diff --git a/docs/adr/0056-verify-ui-with-linux-side-playwright-and-attach-produced-demo-evidence-to-prs.md b/docs/adr/0056-verify-ui-with-linux-side-playwright-and-attach-produced-demo-evidence-to-prs.md index e54af012b0..f08d175c13 100644 --- a/docs/adr/0056-verify-ui-with-linux-side-playwright-and-attach-produced-demo-evidence-to-prs.md +++ b/docs/adr/0056-verify-ui-with-linux-side-playwright-and-attach-produced-demo-evidence-to-prs.md @@ -213,6 +213,37 @@ produced video and screenshots on the PR from the user's own gh login. - Vision review stays a source of leads until the planted-defect eval measures it; the layer order stays judgment until then. +## Findings, 2026-10-11 + +Probes of the planted-defect eval on the `eval-browser-dispatch` lane (Claude Code 2.1.296, +GitHub-hosted ubuntu-24.04) leave the decision unchanged but constrain how it is measured. Runs are +in `melodic-software/claude-code-plugins` Actions. + +- **The eval hides a case's own files from the agent.** `Read` of the case's canary file is refused + and the browser cannot load it; the `evals-hidden-from-read` and `evals-hidden-from-browser` + graders pass (runs 38146517909, 38151389439). +- **Narrow Bash grants match as written.** An `--allow-tools` grant such as + `Bash(playwright-cli -s=* goto *)` produced no permission refusal (run 38146517909). +- **`playwright-cli` cannot run inside `claude plugin eval` Bash on Linux.** Chromium aborts with + `FATAL:chrome/browser/process_singleton_posix.cc:297] Check failed: . socket() failed: Operation not permitted (1)` + (run 38150050988): the sandbox's seccomp filter blocks `socket(AF_UNIX)`, and + the CLI's own daemon also uses a Unix socket on Linux. The eval writes its own sandbox settings; + case keys, CLI flags and a managed policy setting `sandbox.network.allowAllUnixSockets: true` did + not lift the block (run 38151389439). Pointers: + [plugin evals](https://code.claude.com/docs/en/plugin-evals) and the `allowAllUnixSockets` entry + in the [settings reference](https://code.claude.com/docs/en/settings-reference). As of: + 2026-10-11. Recheck trigger: a Claude Code release that exposes sandbox network options to eval + runs. +- **`CLAUDE_CODE_SUBPROCESS_ENV_SCRUB=1` breaks every eval Bash call.** With it set, a one-command + `echo` case fails with + `bwrap: Can't create file at /home/cwd/.eval-artifacts: Read-only file system`; with it `0`, the same case runs (runs 38114987035 and 38118268413). Observed on the + versions above and not filed upstream. + +Consequences: a measured browser eval needs an MCP-driven harness +([#7137](https://github.com/melodic-software/claude-code-plugins/issues/7137)). The rubric's tool +choices stand, and any catch rate or other measured figure states which path, CLI or MCP, it +covers. The Claude in Chrome from WSL open item below is untouched by these runs. + ## Open items - **Claude in Chrome from WSL.** A live test of `claude --chrome` from WSL pairing with the user's @@ -226,5 +257,5 @@ produced video and screenshots on the PR from the user's own gh login. decision, and a probe; the docs cover it for VMs, not WSL. Recheck this record when cli/cli#14309 closes or gh's attachment token allowlist changes, when -`@playwright/cli` or Playwright releases zoom or idle trimming, or when Anthropic's chrome page -changes its WSL note. +`@playwright/cli` or Playwright releases zoom or idle trimming, when Anthropic's chrome page +changes its WSL note, or when `claude plugin eval` lets a run open Unix sockets. diff --git a/plugins/testing/skills/run-e2e/context/e2e.md b/plugins/testing/skills/run-e2e/context/e2e.md index 7662691a28..5dac6792bb 100644 --- a/plugins/testing/skills/run-e2e/context/e2e.md +++ b/plugins/testing/skills/run-e2e/context/e2e.md @@ -101,6 +101,18 @@ baselines). Recheck trigger: a benchmark measures current vision models on UI de injected-defect eval in this repository reports catch rates per check, or either cited Playwright page changes what it says about automated accessibility coverage or same-environment baselines. +No catch rate per check is measured yet. On Linux, `playwright-cli` cannot run inside +`claude plugin eval`'s Bash: the eval sandbox blocks Unix sockets, so Chromium aborts at launch, and +no case key, CLI flag or managed `sandbox.network.allowAllUnixSockets` setting lifted the block +(Claude Code 2.1.296, eval-lane runs 38150050988 and 38151389439; findings in the marketplace repository's ADR 0056). Browser +eval cases therefore move to an MCP-driven harness +([#7137](https://github.com/melodic-software/claude-code-plugins/issues/7137)). The rubric rows +above do not change; a catch rate quoted here names the path it measured, CLI or MCP. Pointers: +[plugin evals](https://code.claude.com/docs/en/plugin-evals) and the `allowAllUnixSockets` entry in +the [settings reference](https://code.claude.com/docs/en/settings-reference). As of: 2026-10-11. +Recheck trigger: a Claude Code release that exposes sandbox network options to eval runs, or #7137 +reports catch rates. + ### Recording tier (optional) Recording is off by default. The screenshot evidence above is the floor. When the `recording` key ([e2e-config.md](e2e-config.md)) is set, a run also captures a moving record; it supplements the screenshots, never replaces them. A trace is not a recording tier: every UI drive starts one before the first step (workflow step 5), whatever `recording` says, so a failed scenario carries one. From 1021713f761a4fb222d9337ccf80f5deaf99614a Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sun, 11 Oct 2026 11:37:44 -0400 Subject: [PATCH 2/3] docs(testing): say which Unix-socket remedies were tried Only the managed allowAllUnixSockets policy was tried (run 38151389439). Case keys and CLI flags were not tried; plugin evals document none. Co-Authored-By: Claude Opus 5.5 --- ...de-playwright-and-attach-produced-demo-evidence-to-prs.md | 5 +++-- plugins/testing/skills/run-e2e/context/e2e.md | 3 ++- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/adr/0056-verify-ui-with-linux-side-playwright-and-attach-produced-demo-evidence-to-prs.md b/docs/adr/0056-verify-ui-with-linux-side-playwright-and-attach-produced-demo-evidence-to-prs.md index f08d175c13..080bb00140 100644 --- a/docs/adr/0056-verify-ui-with-linux-side-playwright-and-attach-produced-demo-evidence-to-prs.md +++ b/docs/adr/0056-verify-ui-with-linux-side-playwright-and-attach-produced-demo-evidence-to-prs.md @@ -228,8 +228,9 @@ in `melodic-software/claude-code-plugins` Actions. `FATAL:chrome/browser/process_singleton_posix.cc:297] Check failed: . socket() failed: Operation not permitted (1)` (run 38150050988): the sandbox's seccomp filter blocks `socket(AF_UNIX)`, and the CLI's own daemon also uses a Unix socket on Linux. The eval writes its own sandbox settings; - case keys, CLI flags and a managed policy setting `sandbox.network.allowAllUnixSockets: true` did - not lift the block (run 38151389439). Pointers: + plugin evals expose no case key or CLI flag for sandbox network options (plugin-evals docs, as of + 2026-10-11), and a managed `sandbox.network.allowAllUnixSockets: true` policy did not lift the + block (run 38151389439). Pointers: [plugin evals](https://code.claude.com/docs/en/plugin-evals) and the `allowAllUnixSockets` entry in the [settings reference](https://code.claude.com/docs/en/settings-reference). As of: 2026-10-11. Recheck trigger: a Claude Code release that exposes sandbox network options to eval diff --git a/plugins/testing/skills/run-e2e/context/e2e.md b/plugins/testing/skills/run-e2e/context/e2e.md index 5dac6792bb..fbf412750d 100644 --- a/plugins/testing/skills/run-e2e/context/e2e.md +++ b/plugins/testing/skills/run-e2e/context/e2e.md @@ -103,7 +103,8 @@ page changes what it says about automated accessibility coverage or same-environ No catch rate per check is measured yet. On Linux, `playwright-cli` cannot run inside `claude plugin eval`'s Bash: the eval sandbox blocks Unix sockets, so Chromium aborts at launch, and -no case key, CLI flag or managed `sandbox.network.allowAllUnixSockets` setting lifted the block +plugin evals expose no case key or CLI flag for sandbox network options (plugin-evals docs, as of +2026-10-11), and a managed `sandbox.network.allowAllUnixSockets: true` policy did not lift the block (Claude Code 2.1.296, eval-lane runs 38150050988 and 38151389439; findings in the marketplace repository's ADR 0056). Browser eval cases therefore move to an MCP-driven harness ([#7137](https://github.com/melodic-software/claude-code-plugins/issues/7137)). The rubric rows From f29f6dacc4b249d256bcf7c14d1a03d44925fb02 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sun, 11 Oct 2026 11:45:19 -0400 Subject: [PATCH 3/3] docs(testing): keep eval diagnostics out of the run-e2e skill body Skill bodies point at sources instead of restating volatile specifics. Replace the sandbox and run-ID diagnostics in the run-e2e render-check note with the rule only; the diagnostics stay in ADR 0056. Co-Authored-By: Claude Opus 5.5 --- .../testing/docs-browser-rubric-p7-5c1e9a37.md | 2 +- plugins/testing/skills/run-e2e/context/e2e.md | 15 +++------------ 2 files changed, 4 insertions(+), 13 deletions(-) diff --git a/.changes/testing/docs-browser-rubric-p7-5c1e9a37.md b/.changes/testing/docs-browser-rubric-p7-5c1e9a37.md index 567d1a7655..8d4a8e3848 100644 --- a/.changes/testing/docs-browser-rubric-p7-5c1e9a37.md +++ b/.changes/testing/docs-browser-rubric-p7-5c1e9a37.md @@ -4,4 +4,4 @@ bump: patch ### Changed -- `/testing:run-e2e`'s render checks say no per-check catch rate is measured yet: `playwright-cli` cannot run inside `claude plugin eval`'s Bash on Linux, so browser eval cases move to an MCP-driven harness (#7137), and a quoted catch rate names the path it measured, CLI or MCP. The browser-tool rubric rows are unchanged. +- `/testing:run-e2e`'s render checks say no per-check catch rate is measured yet (measured browser evals are tracked in #7137) and that a quoted catch rate names the harness path it measured, CLI or MCP. The browser-tool rubric rows are unchanged. diff --git a/plugins/testing/skills/run-e2e/context/e2e.md b/plugins/testing/skills/run-e2e/context/e2e.md index fbf412750d..e5e7a48c44 100644 --- a/plugins/testing/skills/run-e2e/context/e2e.md +++ b/plugins/testing/skills/run-e2e/context/e2e.md @@ -101,18 +101,9 @@ baselines). Recheck trigger: a benchmark measures current vision models on UI de injected-defect eval in this repository reports catch rates per check, or either cited Playwright page changes what it says about automated accessibility coverage or same-environment baselines. -No catch rate per check is measured yet. On Linux, `playwright-cli` cannot run inside -`claude plugin eval`'s Bash: the eval sandbox blocks Unix sockets, so Chromium aborts at launch, and -plugin evals expose no case key or CLI flag for sandbox network options (plugin-evals docs, as of -2026-10-11), and a managed `sandbox.network.allowAllUnixSockets: true` policy did not lift the block -(Claude Code 2.1.296, eval-lane runs 38150050988 and 38151389439; findings in the marketplace repository's ADR 0056). Browser -eval cases therefore move to an MCP-driven harness -([#7137](https://github.com/melodic-software/claude-code-plugins/issues/7137)). The rubric rows -above do not change; a catch rate quoted here names the path it measured, CLI or MCP. Pointers: -[plugin evals](https://code.claude.com/docs/en/plugin-evals) and the `allowAllUnixSockets` entry in -the [settings reference](https://code.claude.com/docs/en/settings-reference). As of: 2026-10-11. -Recheck trigger: a Claude Code release that exposes sandbox network options to eval runs, or #7137 -reports catch rates. +No per-check catch rate is measured yet; measured browser evals are tracked in +[#7137](https://github.com/melodic-software/claude-code-plugins/issues/7137). Any catch rate quoted +here names the harness path it measured, CLI or MCP. ### Recording tier (optional)