diff --git a/docs/probes/2026-09-16-skills-discovery/.gitignore b/docs/probes/2026-09-16-skills-discovery/.gitignore new file mode 100644 index 0000000000..330a7f74d9 --- /dev/null +++ b/docs/probes/2026-09-16-skills-discovery/.gitignore @@ -0,0 +1,3 @@ +out/ +*.stderr.log +*-tui.log diff --git a/docs/probes/2026-09-16-skills-discovery/capability-matrix.md b/docs/probes/2026-09-16-skills-discovery/capability-matrix.md new file mode 100644 index 0000000000..c230b10e76 --- /dev/null +++ b/docs/probes/2026-09-16-skills-discovery/capability-matrix.md @@ -0,0 +1,40 @@ +# Capability matrix — repo-local skill discovery + +Generated by `report.py` from `matrix.json`; do not edit by hand. A `—` cell means the entry +could not run (see its status line below and `findings.md`). + +| Entry | Version tested | Modes | Native root | Roots consumed | Startup mechanism proven | Exclusion preserving load | Vendor-isolated destination | Reload path | Live catalogue | Startup rewrite | Resume | Nested cwd | Worktree | Peer hook | Interactive | GUI status | Minimum version | +| -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | +| agy | 1.2.4, 1.2.5 | print | .agents/skills | — | — | — | — | — | — | — | — | — | — | — | — | Antigravity IDE: untested (manual procedure in findings.md) | — | +| agy-clidir | 1.2.4 | print | .agents/skills | .agent/skills, .agents/skills | n/a (not run) | gitignore, info-exclude | .agent/skills | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | Antigravity IDE: untested (manual procedure in findings.md) | 1.2.4 | +| agy-dirlayout | 1.2.4, 1.2.5 | print, tui | .agents/skills | .agent/skills, .agents/skills | none | gitignore, info-exclude | .agent/skills | none | n/a (not run) | delete=stale; update=visible_first_turn | add=visible_first_turn; update=visible_first_turn | ancestor=visible_first_turn; local=visible_first_turn | linked-other=not_visible; linked-own=visible_first_turn | n/a (not run) | S1=visible_first_turn; hook-adds-skill=not_visible; add=visible_live | Antigravity IDE: untested (manual procedure in findings.md) | 1.2.4 | +| claude | 2.1.273 (Claude Code), 2.1.274 (Claude Code) | print, tui | .claude/skills | .claude/skills | --settings file over the real config root hooks.SessionStart (tui) | gitignore, info-exclude | none | none | n/a (not run) | delete=revoked; update=stale | add=visible_first_turn | ancestor=visible_first_turn; local=visible_first_turn | linked-other=not_visible; linked-own=visible_first_turn | n/a (not run) | S1=visible_first_turn; hook-adds-skill=visible_first_turn; add=not_visible | n/a | 2.1.273 (Claude Code) | +| codex | codex-cli 0.154.0 | daemon, print, tui | .agents/skills | .agents/skills, .codex/skills | none | gitignore, info-exclude | none | none | add=visible_live; delete=revoked; update=visible_live | delete=not_visible/stale; update=visible_first_turn | add=visible_first_turn; update=visible_first_turn | ancestor=visible_first_turn; local=visible_first_turn | linked-other=not_visible; linked-own=visible_first_turn | hook-from-peer=visible_live | S1=visible_first_turn; hook-adds-skill=not_visible; add=visible_live | n/a | codex-cli 0.154.0 | +| copilot | GitHub Copilot CLI 1.0.85. | daemon, print, tui | .github/skills | .agents/skills, .claude/skills, .github/skills | none | gitignore, info-exclude | .github/skills | /skills reload | add=not_visible; delete=stale; update=stale | delete=revoked; update=visible_first_turn | add=visible_first_turn; update=stale | ancestor=visible_first_turn; local=visible_first_turn | linked-other=not_visible; linked-own=visible_first_turn | hook-from-peer=not_visible | S1=visible_first_turn; hook-adds-skill=not_visible; add=not_visible; reload=visible_after_reload | n/a | GitHub Copilot CLI 1.0.85. | +| cursor | 2026.09.15-d2fe57e | daemon, print, tui | .cursor/skills | .agents/skills, .claude/skills, .codex/skills, .cursor/skills | project .cursor/hooks.json workspaceOpen pluginPaths (print) | gitignore, info-exclude | .cursor/skills | project .cursor/hooks.json workspaceOpen pluginPaths | add=not_visible; delete=revoked; update=visible_live | delete=revoked; update=visible_first_turn | add=not_visible; update=visible_first_turn | ancestor=not_visible; local=visible_first_turn | linked-other=not_visible; linked-own=visible_first_turn | hook-from-peer=not_visible | S1=visible_first_turn; hook-adds-skill=not_visible | Cursor desktop: untested (manual procedure in findings.md) | 2026.09.15-d2fe57e | +| cursor-userhooks | 2026.09.15-d2fe57e | daemon, print | .cursor/skills | n/a (not run) | user ~/.cursor/hooks.json workspaceOpen (print) | n/a (not run) | n/a (not run) | user ~/.cursor/hooks.json workspaceOpen | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a | 2026.09.15-d2fe57e | +| gemini | 0.60.0 | daemon, print, tui | .gemini/skills | — | — | — | — | — | — | — | — | — | — | — | — | n/a | — | +| kiro | kiro-cli 2.21.4 | daemon, print, tui | .kiro/skills | .kiro/skills | agent hooks.agentSpawn (cli 2.x) (daemon); agent hooks.agentSpawn (cli 2.x) (print); agent hooks.agentSpawn (cli 2.x) (tui) | gitignore, info-exclude | .kiro/skills | none | add=visible_live; delete=revoked; update=visible_live | delete=revoked; update=visible_first_turn | add=visible_first_turn; update=visible_first_turn | ancestor=not_visible; local=visible_first_turn | linked-other=not_visible; linked-own=visible_first_turn | hook-from-peer=visible_live | S1=visible_first_turn; hook-adds-skill=visible_first_turn; add=visible_live | Kiro IDE: untested (manual procedure in findings.md) | kiro-cli 2.21.4 | +| kiro-agent-bare | kiro-cli 2.21.4 | daemon, print | .kiro/skills | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a | kiro-cli 2.21.4 | +| kiro-agent-skills | kiro-cli 2.21.4 | daemon, print | .kiro/skills | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a | kiro-cli 2.21.4 | +| opencode-v1 | 1.18.31 | daemon, print, tui | .opencode/skills | .agents/skills, .claude/skills, .opencode/skills | plugin event session.created (print) | gitignore, info-exclude | .opencode/skills | none | add=not_visible; delete=stale; update=stale | delete=revoked; update=stale/visible_first_turn | add=visible_first_turn; update=visible_first_turn | ancestor=visible_first_turn; local=visible_first_turn | linked-other=not_visible; linked-own=visible_first_turn | hook-from-peer=not_visible | S1=visible_first_turn; add=not_visible | n/a | 1.18.31 | +| opencode-v2 | opencode v2.0.4 | daemon, print, tui | .opencode/skills | .agents/skills, .claude/skills, .opencode/skills | n/a (not run) | gitignore, info-exclude | .opencode/skills | n/a (not run) | add=not_visible; delete=revoked; update=stale | n/a (not run) | add=visible_first_turn; update=visible_first_turn | ancestor=visible_first_turn; local=visible_first_turn | linked-other=not_visible; linked-own=visible_first_turn | n/a (not run) | S1=visible_first_turn; add=visible_live | n/a | opencode v2.0.4 | +| pi | 0.85.1 | daemon, print, tui | .pi/skills | .agents/skills, .pi/skills | extension session_start + resources_discover skillPaths (daemon); extension session_start + resources_discover skillPaths (print) | gitignore, info-exclude | .pi/skills | extension session_start + resources_discover skillPaths | add=not_visible; delete=not_visible; update=visible_live | delete=revoked; update=visible_first_turn | add=visible_first_turn; update=visible_first_turn | ancestor=not_visible; local=visible_first_turn | linked-other=not_visible; linked-own=visible_first_turn | hook-from-peer=not_visible | S1=visible_first_turn; hook-adds-skill=not_visible; add=not_visible | n/a | 0.85.1 | + +## Status per entry + +- `agy`: S1 failed: the native root's skill was not loaded +- `agy-clidir`: measured +- `agy-dirlayout`: measured +- `claude`: measured +- `codex`: measured +- `copilot`: measured +- `cursor`: measured +- `cursor-userhooks`: measured +- `gemini`: untested: exception=RuntimeError("vendor exit 1 with no reply; extract failed: JSONDecodeE; exception=RuntimeError('no reply read from the screen; exit None; timeout no PRO; exception=RuntimeError('vendor exit -9 with no reply; session/new failed: {"json +- `kiro`: measured +- `kiro-agent-bare`: measured +- `kiro-agent-skills`: measured +- `opencode-v1`: measured +- `opencode-v2`: measured +- `pi`: measured diff --git a/docs/probes/2026-09-16-skills-discovery/findings.md b/docs/probes/2026-09-16-skills-discovery/findings.md new file mode 100644 index 0000000000..bbbdc8ffcd --- /dev/null +++ b/docs/probes/2026-09-16-skills-discovery/findings.md @@ -0,0 +1,431 @@ +# Repo-local skill discovery probes — 2026-09-16 + +**Subject:** Claude Code 2.1.273 (native installer, `~/.local/bin/claude`), codex-cli 0.154.0 (brew cask), +Gemini CLI 0.60.0 (npm `@google/gemini-cli`), Pi 0.85.1 (npm `@earendil-works/pi-coding-agent`), +GitHub Copilot CLI 1.0.85 (npm `@github/copilot`), kiro-cli 2.21.4 (brew cask), OpenCode 1.18.31 (npm +`opencode-ai`), OpenCode 2.0.4 (npm `@opencode/cli`, prefix `~/.local/opencode-v2`), cursor-agent +2026.09.15-d2fe57e (vendor installer), agy 1.2.4 (vendor installer). macOS 26.6.2 / arm64. +**Driver:** `probe.py` (scenarios S0–S4, two runs per arm, a third on disagreement, one single-run +confirmation per root the multi prompt missed); `report.py` renders `capability-matrix.md` from +`matrix.json`; `selftest.py` covers the kit with no vendor binary. +**Cost:** the free phase issues zero model requests. The recorded matrix holds 803 model turns across +both passes; re-measurements while adapters were being corrected added roughly a hundred more. + +Every verdict below is the starting session's own reply. A skill counts as *discovered* when the model +names it and as *loaded* when the reply carries the token that exists only in the skill body. A file on +disk, a hook exit code or a listing command never decides a row. Print mode is the vendor's headless +single-prompt launch; daemon mode is the launch kcap's daemon uses (ACP for Copilot, Gemini, Kiro, +Cursor and OpenCode, `codex app-server`, `pi --mode rpc`); interactive launches drive the vendor's own +UI on a pseudo-terminal and are measured in pass 2, below. + +## How to reproduce + +``` +python3 docs/probes/2026-09-16-skills-discovery/selftest.py # kit self-tests, no vendors +python3 docs/probes/2026-09-16-skills-discovery/probe.py --harness claude --mode print # free phase +python3 docs/probes/2026-09-16-skills-discovery/probe.py --harness claude --mode print --turn # full sweep +python3 docs/probes/2026-09-16-skills-discovery/probe.py --emit && python3 docs/probes/2026-09-16-skills-discovery/report.py +``` + +A sweep resumes: arms with settled rows are not re-spent, partial arms are topped up, and an arm with an +`untested` row is measured again. `--rerun` clears an entry's arms first. Run one vendor at a time: +two concurrent `cursor-agent` processes made its print mode fail mid-turn. Environment switches: +`KCAP_PROBE_CLAUDE_REAL_CONFIG=0|1` forces the Claude config-root mode, `GEMINI_API_KEY` makes Gemini +runnable on an account whose OAuth tier is refused, `KIRO_API_KEY` is passed through for Kiro, +`KCAP_PROBE_OPENCODE_MODEL` pins the OpenCode model, `KCAP_OPENCODE_V2_PATH` names the V2 binary. +Logins that open a browser are the operator's: `cursor-agent login`, `kiro-cli login`, +`opencode auth login`, `agy` (first launch). + +Every run leaves `out/////run.json` (the record), `run.raw.txt` +(the vendor's full event stream) and `run..stderr.log`; `out/` is git-ignored, `matrix.json` +is the committed roll-up. + +Three entries cannot be isolated in a private home because their login lives in the OS keyring keyed by +the real one: Claude (config directory), Cursor and agy (HOME). They run against the real home with +`KCAP_SKIP=1` so kcap's own hooks stand down, with the probe hook placed in the sandbox repository +(project-level `.cursor/hooks.json`, workspace `.agents/plugins/probe/`) or supplied by flag +(`--settings`); the two variants that must write into the real home (`cursor-userhooks`, +`agy-clidir`) back the file up and restore it after the turn. + +## Claude Code 2.1.273 (print mode; daemon launches use the same `-p`) + +- **Isolation.** The OAuth login lives in the macOS keychain keyed by the config root: an isolated + `CLAUDE_CONFIG_DIR` reports `loggedIn: false`, and so does naming the default root explicitly. The + adapter runs against the real config root with `CLAUDE_CONFIG_DIR` unset, supplies the probe hook + through `--settings `, limits sources with `--setting-sources project` so no user-level plugin + loads, and sets `KCAP_SKIP=1`. `--allowedTools Skill` leaves the model no way to read files, so a + token can only arrive through the Skill tool. +- **S1** `visible_first_turn` ×2. +- **S2** `hook-creates-root` `not_visible` ×2 and `hook-adds-skill` `not_visible` ×2. The SessionStart + hook fired about 0.7 s after launch and the file existed when the reply was written; the model answered + `NO-SKILL`, and with a pre-existing sibling it listed only that sibling. Skills are indexed before + SessionStart hooks run, and the documented watched-root refresh does not reach the first request. +- **S3** both exclusions `visible_first_turn` ×2; `git status` stays clean under both. +- **S4** consumes `.claude/skills` only. +- **Limitation.** No registration or reload mechanism for a headless launch. `.claude/skills` is also + read by Copilot, Cursor and OpenCode, so it is not a vendor-isolated destination. + +## Codex 0.154.0 (print `codex exec`; daemon `codex app-server`) + +- **Isolation.** `CODEX_HOME` with `auth.json` copied keeps the ChatGPT login; `config.toml` carries a + `[projects.""] trust_level = "trusted"` entry. `exec` runs with + `--dangerously-bypass-hook-trust`; the app-server driver reads `hooks/list` (hooks are grouped under + `result.data[].hooks`) and restarts with `-c hooks.state={…trusted_hash…}` to seed trust, as kcap's + daemon does. +- **Native load is a file read.** Codex lists a skill with its path and the model runs + `cat /SKILL.md`. A prompt forbidding "any tool" made Codex answer `NO-SKILL` for a listed skill, + so the prompt forbids searching instead and the adapter records `skill_reads` apart from `searches`. +- **S1** `visible_first_turn` ×2 in both modes, `skill_reads=1 searches=0`. +- **S2** both hook arms `not_visible` ×2 in both modes; the hook fired (`hook_trust=seeded` in daemon + mode) and the file existed, the first request did not list it. +- **S3** both exclusions `visible_first_turn` ×2 in both modes. +- **S4** consumes `.agents/skills` and repo-local `.codex/skills`; the latter is read by Cursor too, so + Codex has no vendor-isolated destination either. + +## Pi 0.85.1 (print `pi -p --mode json`; daemon `pi --mode rpc`) + +- **Isolation.** `PI_CODING_AGENT_DIR` with `auth.json`, `models.json`, `settings.json` copied; auth + through the `openai-codex` provider. `--approve` trusts the sandbox repo's project resources. The + native load is a `read` of the listed SKILL.md path (`skill_reads`). +- **S1** `visible_first_turn` ×2 in both modes. +- **S2** `hook-creates-root` and `hook-adds-skill` (an extension whose `session_start` runs the hook + script) `not_visible` ×2 in both modes. **`registration` `visible_after_reload` ×2 in both modes:** an + extension that writes the skill in `session_start` and returns the skills root from + `resources_discover` (`{skillPaths: []}`) makes the skill part of the first request. +- **S3** both exclusions `visible_first_turn` ×2 in both modes. +- **S4** consumes `.pi/skills` and `.agents/skills`; `.pi/skills` is Pi's vendor-isolated destination. + +## GitHub Copilot CLI 1.0.85 (print `copilot -p`; daemon `copilot --acp --stdio`) + +- **Isolation.** `COPILOT_HOME` with `COPILOT_GITHUB_TOKEN` from `gh auth token` (an OAuth `gho_` + token). The native load is Copilot's own `skill` tool (`skill_loads`). +- **S1** `visible_first_turn` ×2 in both modes; `copilot skill list --json` lists the probe skill too. +- **S2** both hook arms `not_visible` ×2 in both modes. The `sessionStart` hook in + `$COPILOT_HOME/hooks/*.json` fired about 3 s after launch; the skill was not in the first request. +- **S3** both exclusions `visible_first_turn` ×2 in both modes. +- **S4** consumes `.github/skills`, `.agents/skills` and `.claude/skills`, all documented. + `.github/skills` is Copilot's vendor-isolated destination. +- **Limitation.** `/skills reload` is interactive only; no headless reload or registration path. + +## Kiro CLI 2.21.4 (print `kiro-cli chat --no-interactive`; daemon `kiro-cli acp`) + +- **Isolation.** `KIRO_HOME`; the IAM Identity Center login lives outside it and survives. The brew cask + ships the launcher only: `kiro-cli-chat` and `kiro-cli-term` had to be linked from the app bundle + into `~/.local/bin`. This build is the 2.x hook generation, so the hook is `hooks.agentSpawn` inside a + custom agent that is made the default; that agent must be cloned from `kiro_default` + (`kiro-cli agent create --from kiro_default`, as kcap's installer does), because a minimal agent + definition has no tools and the model then cannot read the listed file ("I don't have the ability to + actually invoke tools"). The native load is a `read` of the listed SKILL.md path. +- **S1** `visible_first_turn` ×2 in both modes. +- **S2 `hook-creates-root` and `hook-adds-skill` `visible_first_turn` ×2 in both modes.** `agentSpawn` + fires when the session is created, about 2 s before the first prompt in ACP mode, and the skill + catalogue is built afterwards: a plain file drop from the startup hook reaches the first request. +- **S3** both exclusions `visible_first_turn` ×2 in both modes. +- **S4** consumes `.kiro/skills` only; it is Kiro's vendor-isolated destination. +- **Custom agents.** `kiro-agent-bare` (cloned default agent with `resources: []`) and + `kiro-agent-skills` (cloned agent declaring `skill://.kiro/skills/*/SKILL.md`) both `visible_first_turn` + ×2 in both modes: a cloned agent inherits the default skills whether or not it declares them. +- **Headless without a login** opens a browser sign-in instead of answering; the adapter checks + `kiro-cli whoami` first. + +## OpenCode 1.18.31 (print `opencode run --format json`; daemon `opencode acp`) + +- **Isolation.** `OPENCODE_CONFIG_DIR` plus `XDG_DATA_HOME` pointed at the sandbox, with + `~/.local/share/opencode/auth.json` (the `github-copilot` provider) copied. The native load is the + `skill` tool (`skill_loads`); in ACP mode the tool call is titled `Loaded skill: `. +- **S1** `visible_first_turn` ×2 in both modes. +- **S2 print:** `hook-adds-skill` **`visible_first_turn` ×2** — a skill written by the plugin's + `session.created` handler into an existing root is in the first request; `hook-creates-root` + `not_visible` ×2 — a root created by the same handler is not; `registration` + (`experimental.chat.system.transform` running the script) `not_visible` ×2. +- **S2 daemon:** all three arms `not_visible` ×2 (one run timed out and was re-measured). The ACP session + builds its catalogue before the plugin's write lands. +- **S3** both exclusions `visible_first_turn` ×2 in both modes. +- **S4** consumes `.opencode/skills`, `.agents/skills` and `.claude/skills`, all documented. + +## OpenCode 2.0.4 (print `opencode run --standalone`; daemon `opencode acp`) + +- **Launch.** `--standalone` is accepted only after `run` (before it, the CLI prints its usage; the kit + treats printed usage as a failed run) and not at all for `acp`, which uses the background service; the + adapter stops that service after every turn. Auth and config isolation as for 1.x. +- **S1** `visible_first_turn` ×2 in both modes through the `skill` tool. +- **S2** untested: no local plugin placement loaded in this build — a `Plugin.define` module in the + config directory's `plugins/`, in the project's `.opencode/plugins/`, listed under `plugins` in either + `opencode.json`, or with a `package.json`, never ran its `setup` (a marker-writing plugin left no + marker at service start or during a `run`), and `opencode plugin add ` fails with a stack + trace. The `setup`, `skill.reload()` and `prompt`-hook arms wait for a working local plugin route. +- **S3** both exclusions `visible_first_turn` ×2 in both modes. +- **S4** consumes `.opencode/skills`, `.agents/skills` and `.claude/skills`, as 1.x does; the multi + prompt was inconsistent across its three passes and every root was settled by its own confirmation. +- The `opencode-v2-prompt` entry (an awaited `prompt` hook plus `ctx.skill.reload()`) has no rows: it + cannot run until a local plugin loads. + +## Cursor CLI 2026.09.15 (print `cursor-agent -p`; daemon `cursor-agent acp`) + +- **Isolation.** The login is keyring-held and not found from a private HOME, so the CLI runs against + the real home with `KCAP_SKIP=1`; the probe hook is the sandbox repository's `.cursor/hooks.json` + (`cursor`) or a merge into the real `~/.cursor/hooks.json` restored after the turn + (`cursor-userhooks`). Print mode ends every tool-using turn with `RetriableError: WritableIterable is + closed` and exit 1 after three failed stream reconnects; the `json` and `text` formats then print + nothing, so the adapter reads the answer from `--output-format stream-json`, which has already streamed + it. The native load is a `readToolCall` of the listed SKILL.md path. +- **S1** `visible_first_turn` ×2 in both modes. +- **S2 print:** project-level `sessionStart` fires; `hook-creates-root` `not_visible` ×2 and + `hook-adds-skill` `not_visible` ×2. With the user-level file (`cursor-userhooks`) `hook-adds-skill` + `not_visible` ×2 and **`hook-creates-root` `not_visible` ×2 then `visible_first_turn` once**: a + fire-and-forget hook races Cursor's indexing, and the order decides. `workspaceOpen` does fire in + print mode and its `pluginPaths` route delivers, from the project file and the user file alike: + `registration` is `visible_after_reload` ×2 for both, the hook writing the skill into a plugin + directory it returns. Over ACP it stays `untested`: no Cursor hook fires there at all. +- **S2 daemon:** no hook fires in ACP mode, from the project file or the user file; `sessionStart` and + `workspaceOpen` alike (`untested`, hook never fired). +- **S3** both exclusions `visible_first_turn` ×2 in both modes. +- **S4** consumes `.cursor/skills`, `.agents/skills`, `.claude/skills` and `.codex/skills`, all + documented; `.cursor/skills` is Cursor's vendor-isolated destination. + +## Antigravity CLI 1.2.4 (`agy -p`; no daemon mode) + +- **Launch.** Skills are slash commands that expand in print mode, so the adapter leads the prompt with + `/`; without the slash the model never sees a skill body. The current directory alone is + not the workspace: `--add-dir ` is required before repo-local skills load. The OAuth session is + keyring-held and starts a new browser sign-in from any private HOME, so agy runs against the real home + with `KCAP_SKIP=1`. +- **Layouts.** `.agents/skills//SKILL.md` loads (`agy-dirlayout`, and `agy-clidir`); the flat + `.agents/skills/.md` layout the CLI documentation describes does not (`agy` entry, S1 + `not_visible` ×2), and a flat file under the documented global `~/.gemini/antigravity-cli/skills/` + does not either while a `/SKILL.md` directory there does. +- **S1** `visible_first_turn` ×2 (directory layout). +- **S2** `hook-creates-root` and `hook-adds-skill` `not_visible` ×2: the workspace plugin + `.agents/plugins/probe/hooks.json` `PreInvocation` hook fired about 2 s after launch and the skill was + not listed. The same hook placed in the CLI docs' global `~/.gemini/antigravity-cli/plugins/` never + fired (`agy-clidir`, `untested`). +- **S3** both exclusions `visible_first_turn` ×2. +- **S4** consumes `.agents/skills` and legacy `.agent/skills`, both documented. + +## Gemini CLI 0.60.0 — untested (credential) + +The copied OAuth credential is refused: `IneligibleTierError: This client is no longer supported for +Gemini Code Assist for individuals. To continue using Gemini, please migrate to the Antigravity suite` +(`out/gemini/print/S1/S1_native/run1.gemini.stderr.log`). Print mode exits 1 before any model call and +ACP mode rejects `session/new` with "Gemini API key is missing or not configured". Every turn arm is +`untested` with that reason. The adapter is complete (hooks in `.gemini/settings.json`, +`security.auth.selectedType`, folder trust off, `--approval-mode yolo`, `gemini skills list` as the +catalogue check) and runs as soon as `GEMINI_API_KEY` is exported. + +## Cross-vendor consumption (S4) + +| Root | Claude | Codex | Pi | Copilot | Kiro | OpenCode | Cursor | agy | +| -- | -- | -- | -- | -- | -- | -- | -- | -- | +| `.claude/skills` | yes | no | no | yes | no | yes | yes | no | +| `.agents/skills` | no | yes | yes | yes | no | yes | yes | yes | +| `.codex/skills` | no | yes | no | no | no | no | yes | no | +| `.pi/skills` | no | no | yes | no | no | no | no | no | +| `.github/skills` | no | no | no | yes | no | no | no | no | +| `.kiro/skills` | no | no | no | no | yes | no | no | no | +| `.opencode/skills` | no | no | no | no | no | yes | no | no | +| `.cursor/skills` | no | no | no | no | no | no | yes | no | +| `.agent/skills` | no | no | no | no | no | no | no | yes | +| `.gemini/skills` | no | no | no | no | no | no | no | no | + +Every "no" was confirmed with a single-root turn after the multi-prompt run missed it. Vendor-isolated +destinations: `.pi/skills`, `.github/skills`, `.kiro/skills`, `.opencode/skills`, `.cursor/skills`. +Claude and Codex have none: `.claude/skills` is read by three others, `.codex/skills` by Cursor. + +## Git exclusion (S3) + +Every measured entry loads a skill under both `.gitignore` and `info/exclude`, in every mode, with +`git status --porcelain` empty for the skill path and non-empty in the untracked control. Both +mechanisms keep generated skills out of the index without hiding them from any harness measured. + +## Startup ordering (S2) + +| Entry | Hook fires | Hook-written skill in first request | Registration route | +| -- | -- | -- | -- | +| Claude | yes | no | none | +| Codex (print, app-server) | yes | no | none | +| Pi (print, rpc) | yes | no | `resources_discover` skillPaths: yes | +| Copilot (print, acp) | yes | no | none | +| Kiro (print, acp) | yes, at session creation | **yes** | not needed | +| OpenCode 1.x print | yes | only into an existing root | `chat.system.transform`: no | +| OpenCode 1.x acp | yes | no | no | +| OpenCode 2.x | no plugin loads | untested | untested | +| Cursor print | `sessionStart` yes, `workspaceOpen` yes | race (1 of 3) | `workspaceOpen` `pluginPaths` | +| Cursor acp | no | untested | untested | +| agy | workspace `PreInvocation` yes, global plugin no | no | none | + +## Consequences for #778 and #962 + +- A file drop from a startup hook reaches the first request only on Kiro (whose `agentSpawn` runs at + session creation, before the catalogue is built) and, for an existing root, on OpenCode 1.x print + mode. Everywhere else skills are indexed first, and on Cursor the two race. +- Two vendors take a skill handed to them at startup rather than found on disk: Pi, through an + extension that writes in `session_start` and returns the directory from `resources_discover`, and + Cursor in print mode, through a `workspaceOpen` hook that returns `pluginPaths`. Both are + `visible_after_reload`. No headless registration exists for Claude, Codex, Copilot or agy, so their + delivery must happen before launch. +- `info/exclude` and `.gitignore` are both safe for every measured harness. +- Vendor-isolated destinations exist for Pi, Copilot, Kiro, OpenCode, Cursor and Antigravity + (`.agent/skills`); not for Claude or Codex. Two entries of one vendor count as one consumer. +- Native loading differs: Claude, Copilot and OpenCode load through a tool of their own; Codex, Pi, + Kiro and Cursor read the listed file; agy expands a slash command. A prompt that forbids tool use + defeats the file readers, so the kit forbids searching instead and records listed reads apart from + searches. +- Layout matters for agy: only `/SKILL.md` loads, the documented flat file does not; and the + repository must be added to the workspace. + +## Manual GUI procedures + +For Cursor desktop, Antigravity IDE and Kiro IDE, which this kit cannot launch: + +1. Create a fresh git repository with one commit and open it in the IDE with the vendor's startup hook + configured to run `probe-hook.sh` (generate one with `lib/hook_script.py`, or copy the shape from the + matching adapter's `install_startup_hook`). +2. In the first prompt of the first session, paste the single prompt from `lib/probe_skill.py` + (`single_prompt`) for the skill the hook writes (for Antigravity, lead with `/`). +3. The reply decides: the token means `visible_first_turn`; the skill's name without the token means + `catalogue_only`; `NO-SKILL` means `not_visible`. Repeat once. Record the IDE version and the reply + under `out//interactive/` by hand. + +## Kit traps found during pass 1 + +- Claude's keychain login is keyed by the config root; any explicit `CLAUDE_CONFIG_DIR` loses it. + Cursor's and agy's are keyed by HOME, and a signed-out agy starts a browser sign-in on every launch. +- A prompt that says "do not run any tool" makes Codex, Pi, Kiro and Cursor answer `NO-SKILL` for a + listed skill. +- The app-server's `hooks/list` nests hooks under `result.data[].hooks`. +- Gemini re-execs itself, and asyncio's `Process.wait()` never returns while the grandchild holds + stdout: children run in their own session and the session is killed on the graceful-exit timeout. +- A vendor that exits without a reply (Gemini's tier refusal, Cursor's stream error) must be `untested`, + never `not_visible`; a printed usage text is a failed run, not a negative control. +- A reply that echoes another skill's name beside `NO-SKILL` is a negative, not `catalogue_only`; a + Kiro agent without tools names the listed skill and cannot read it, which is a tool-access confound. +- Two `cursor-agent` processes at once make print mode fail mid-turn. + +# Pass 2 — lifecycle and interactive mode (2026-09-17) + +**Subject:** the same machine and binaries as pass 1, with Claude Code 2.1.274 and agy 1.2.5 (both +auto-updated mid-run, so the matrix carries a row per version). macOS 26.6.2 / arm64. +**Driver:** the same kit with three launch modes. `--mode tui` drives the vendor's interactive UI on a +pseudo-terminal: the reader applies the cursor movements to a grid, because a coding agent streams its +answer beside a spinner that redraws in place and the concatenated bytes interleave the two. + +Pass 1 settled that discovery does not depend on the launch mode: exclusion and root verdicts agreed +between print and daemon mode in 135 of 139 measured pairs. So the discovery-only scenarios stay where +pass 1 measured them, and pass 2 adds the scenarios that need a second turn or a different tree: + +| Scenario | Question | Modes | +| -- | -- | -- | +| S5 live catalogue | does a session see a skill added, rewritten or deleted while it is open? | daemon, tui | +| S6 startup rewrite | does a startup hook that rewrites or deletes an existing skill reach the first request? | print, daemon | +| S7 resume | does a resumed session see what changed after the first one ended? | print | +| S8 nested cwd | launched from a subdirectory, is the repository root's skill found? | print | +| S9 worktrees | in a linked worktree, whose skills are found, under `info/exclude`? | print | +| S10 peer hook | does one session see the skill another session's startup hook wrote? | daemon | + +A two-turn row is decided by the second turn; the first turn's verdict travels in the notes as +`turn1=`. A later turn asks for the token the skill carries *now* and carries a mark of its own, so a +session answering from its own context, or a UI redrawing the earlier answer, is wrong rather than +lucky. New verdicts: `visible_live` (the change reached an open session), `stale` (the old body +survived), `revoked` (`NO-SKILL` after a delete). + +## What a live session sees (S5, S10) + +| Harness | add | rewrite | delete | another session's hook | +| -- | -- | -- | -- | -- | +| Codex | visible_live | visible_live | revoked | visible_live | +| Kiro | visible_live | visible_live | revoked | visible_live | +| Cursor | not visible | visible_live | revoked | not visible | +| Pi | not visible | visible_live | not visible | not visible | +| Copilot | not visible | stale | stale | not visible | +| OpenCode 1.x | not visible | stale | stale | not visible | +| OpenCode 2.x | not visible | stale | revoked | untested (no plugin loads) | + +Two behaviours separate the field. Codex and Kiro re-enumerate the skills directory on every turn, so +a materializer can write into a session that is already running, including from another process. Pi, +Cursor and OpenCode fix the catalogue at session start but read the listed file when the skill is used, +which is why a rewrite lands while an addition does not. Copilot caches both: an open session serves +the body it read at startup. + +In interactive mode the same split holds, with two differences: OpenCode 2.x does see an addition live +(`visible_live`), and Copilot has the only working reload command. `/skills reload` moves its +interactive session from `NO-SKILL` to the token, recorded as `visible_after_reload`. No other harness +has such a command: Codex's palette is `/model /fast /ide /permissions /keymap /vim /experimental +/approve`, Pi exposes skills as `skill:` entries with no reload, Cursor and Antigravity only list. + +## What a startup hook can still change (S6) + +Rewriting an existing skill's body from the startup hook reaches the first request on every harness +that could run the arm except Claude, which keeps serving the body it indexed (`stale`). OpenCode 2.x +is excluded from that statement: its rewrite rows are untested in both modes, because no local plugin +loads. Deleting the skill is honoured +everywhere except Antigravity, which still answers from the deleted skill, and Codex's daemon mode, +which serves the body while the file is already gone. Cursor's daemon mode records `untested`: no +Cursor hook fires over ACP, as pass 1 also found. OpenCode 2.x records `untested` throughout, because +no local plugin loads from any placement. + +Interactive mode changes Claude's answer to the pass 1 question. A skill written by its `SessionStart` +hook *is* visible to the first interactive request (`visible_first_turn`), while the same hook in print +mode is not. Kiro is visible in both, as its `agentSpawn` timing predicts. Codex, Pi, Copilot, Cursor +and Antigravity are not visible in either. + +## Resuming a session (S7) + +Every harness that resumes at all sees a skill added after the first session ended, except Cursor, +which does not. A rewritten body is picked up by all of them except Copilot, which serves the old one. + +Claude is the exception worth naming: its resumed session returns `stop_reason: refusal` with zero +output tokens, twice, for the rewrite question, while the same resume route answers the addition +question normally. The row is `untested` with that reason, not a negative. Kiro has no session id in +its non-interactive output at all, so its resume is per launch directory (`chat --resume`). + +## Where a harness looks (S8, S9) + +Launched from `/sub/dir`, Claude, Codex, Copilot, OpenCode and Antigravity find a skill in the +repository root as well as in the subdirectory. Pi, Kiro and Cursor find only the subdirectory's own. +A materializer that writes to the repository root is therefore invisible to three of nine harnesses +whenever the session starts deeper in the tree. + +In a linked worktree, every harness finds the worktree's own skills under `info/exclude` and none of +them reads the main checkout's. Git resolves a linked worktree's `info/exclude` to the shared common +directory, so one exclusion covers every worktree. + +## Consequences for #778 and #962 + +- **Writing into a live session works only on Codex and Kiro**, and for a rewrite also on Pi, Cursor + and OpenCode. For Copilot the only live route is its reload command, and that exists in the + interactive UI alone. A materializer that must reach an already-running session has to know which of + these three groups the vendor is in. +- **A startup adapter cannot rely on rewriting in place.** Claude serves the old body, so an update + has to land before the process starts. Deleting is safer than rewriting everywhere except + Antigravity and Codex's daemon mode. +- **Resume is the cheapest delivery route** for a skill added between sessions: eight of nine harnesses + pick it up. Claude's refusal makes the rewrite case unmeasurable there, and Cursor ignores additions. +- **Anchor on the launch directory, not the repository root**, or three harnesses will not see the + skill when a session starts in a subdirectory. +- **One `info/exclude` covers a repository and all its worktrees**, and no harness reads a sibling + worktree's skills, so per-worktree materialization is safe and necessary. + +## What the interactive mode cost to drive + +Every one of these was found by replaying the screen the driver keeps beside a failed run: + +- The kit's hook script captured stdin for two seconds to survive a vendor that never closes it. On a + terminal that input belongs to the UI, and OpenCode sat waiting for a keystroke the hook had eaten. + The hook now leaves a tty alone. +- Claude repaints its trust dialog after first drawing it, so an arrow key and the Enter behind it must + be a redraw apart, and the dialog must not be answered until the screen settles. Claude then raises a + browser-tools dialog and, between turns, an auto-mode wizard that takes the next prompt as its input. +- Cursor repaints its whole transcript, so the previous turn's reply is on screen after a clear, and + its composer swallows an Enter sent straight behind the text. +- Codex needs the same hook-trust bypass its headless modes pass, or its startup hook silently does not + run and the arm reads as a harness with no hook at all. +- OpenCode 2.x draws nothing whatsoever while a background service from an earlier run is alive, and + would otherwise serve that run's catalogue. +- OpenCode 1.x's interactive UI never renders with the probe plugin installed: a blank screen for 300 + seconds, twice, while the same driver renders it fine without the plugin. A startup plugin can take + the interactive UI down. +- Cursor's second interactive turn is unmeasurable: it answered once and then twice produced no marked + reply within 300 seconds, so its interactive live-catalogue row stays `untested`. Its daemon mode + answers the same question. +- A run that produces no answer at all is recorded `untested`, never `not_visible`. Claude's refusals + and Codex's rejected resume flags both landed as "the skill was not there" until that rule existed. diff --git a/docs/probes/2026-09-16-skills-discovery/harness/__init__.py b/docs/probes/2026-09-16-skills-discovery/harness/__init__.py new file mode 100644 index 0000000000..47639d379d --- /dev/null +++ b/docs/probes/2026-09-16-skills-discovery/harness/__init__.py @@ -0,0 +1,34 @@ +from __future__ import annotations + +from harness.agy import AgyAdapter, AgyCliDirAdapter, AgyDirLayoutAdapter +from harness.base import Adapter +from harness.claude import ClaudeAdapter +from harness.codex import CodexAdapter +from harness.copilot import CopilotAdapter +from harness.cursor import CursorAdapter, CursorUserHooksAdapter +from harness.fake import FakeAdapter +from harness.gemini import GeminiAdapter +from harness.kiro import KiroAdapter, KiroAgentBareAdapter, KiroAgentSkillsAdapter +from harness.opencode_v1 import OpenCodeV1Adapter +from harness.opencode_v2 import OpenCodeV2Adapter, OpenCodeV2PromptHookAdapter +from harness.pi import PiAdapter + +ENTRIES: dict[str, type[Adapter]] = { + "fake": FakeAdapter, + "claude": ClaudeAdapter, + "codex": CodexAdapter, + "gemini": GeminiAdapter, + "pi": PiAdapter, + "cursor": CursorAdapter, + "cursor-userhooks": CursorUserHooksAdapter, + "copilot": CopilotAdapter, + "kiro": KiroAdapter, + "kiro-agent-bare": KiroAgentBareAdapter, + "kiro-agent-skills": KiroAgentSkillsAdapter, + "opencode-v1": OpenCodeV1Adapter, + "opencode-v2": OpenCodeV2Adapter, + "opencode-v2-prompt": OpenCodeV2PromptHookAdapter, + "agy": AgyAdapter, + "agy-dirlayout": AgyDirLayoutAdapter, + "agy-clidir": AgyCliDirAdapter, +} diff --git a/docs/probes/2026-09-16-skills-discovery/harness/agy.py b/docs/probes/2026-09-16-skills-discovery/harness/agy.py new file mode 100644 index 0000000000..d32e6c7bc9 --- /dev/null +++ b/docs/probes/2026-09-16-skills-discovery/harness/agy.py @@ -0,0 +1,155 @@ +from __future__ import annotations + +import json +import os +import shutil +from pathlib import Path + +from harness.base import Adapter, AskResult, HookInfo, Session +from lib.isolation import Sandbox +from lib.print_driver import print_ask +from lib.probe_skill import NAME_RE + + +class AgyAdapter(Adapter): + """Antigravity CLI. The OAuth session lives in the OS keyring and is not found from a private + HOME (each such launch starts a new sign-in), so agy runs against the real home with kcap's own + hooks stood down, and the probe hook lives inside the sandbox repository as a workspace plugin.""" + + entry = "agy" + harness = "antigravity" + binary = "agy" + lever = "AGY_PROBE_UNUSED" + native_root = ".agents/skills" + documented_roots = frozenset({".agents/skills", ".agent/skills"}) + flat_skill_layout = True + modes = ("print", "tui") + can_resume = True + tui_exit = ("/exit\r", "\x03", "\x03") + + def real_root(self) -> Path | None: + return Path.home() + + def prepare(self, sb: Sandbox) -> None: + sb.env.pop(self.lever, None) + sb.env["HOME"] = os.environ.get("HOME", str(Path.home())) + sb.env["KCAP_SKIP"] = "1" + + def check_auth(self, sb: Sandbox) -> bool | None: + return (Path.home() / ".gemini" / "antigravity-cli" / "settings.json").exists() or None + + def hook_dir(self, sb: Sandbox) -> Path: + return sb.repo / ".agents" / "plugins" / "probe" + + def install_startup_hook(self, sb: Sandbox, script: Path) -> HookInfo: + plugin = self.hook_dir(sb) + plugin.mkdir(parents=True, exist_ok=True) + (plugin / "plugin.json").write_text(json.dumps({"name": "probe", "version": "1.0.0", "description": "probe"}, indent=2) + "\n") + hooks = plugin / "hooks.json" + hooks.write_text(json.dumps({"probe": { + "PreInvocation": [{"type": "command", "command": str(script), "timeout": 15000}], + }}, indent=2) + "\n") + return HookInfo(mechanism="workspace .agents/plugins PreInvocation", config_path=str(hooks)) + + def tui_argv(self, sb: Sandbox) -> list[str] | None: + return [self.binary_path() or self.binary, "--add-dir", str(sb.cwd)] + + def open_session(self, sb: Sandbox, mode: str) -> Session | None: + session = super().open_session(sb, mode) + if session is None: + return None + # Interactive and print mode invoke a skill the same way, so their rows compare. + return _SlashPromptSession(session, lambda: self.cleanup_hook(sb)) + + def session_id(self, res: AskResult) -> str | None: + for line in res.raw.splitlines(): + try: + obj = json.loads(line) + except json.JSONDecodeError: + continue + if isinstance(obj.get("conversation_id"), str): + return obj["conversation_id"] + return None + + def resume(self, sb: Sandbox, session_id: str, prompt: str) -> AskResult | None: + return self.ask(sb, "print", prompt, extra=["--conversation", session_id]) + + def ask(self, sb: Sandbox, mode: str, prompt: str, extra: list[str] = ()) -> AskResult: + def extract(raw: str) -> str: + texts = [] + for line in raw.splitlines(): + try: + obj = json.loads(line) + except json.JSONDecodeError: + continue + if obj.get("event") == "step_update": + su = obj.get("step_update") or {} + if su.get("step_type", "agent_response") == "agent_response" and isinstance(su.get("text_delta"), str): + texts.append(su["text_delta"]) + return "".join(texts) + + # agy exposes skills as slash commands that expand in print mode: invoking the listed + # skill by name is its native load, so the prompt leads with it when it names one. + named = NAME_RE.search(prompt) + text = slash_prefixed(prompt) + # The current directory alone is not the workspace in print mode; --add-dir makes it one. + argv = [self.binary_path() or self.binary, "-p", text, "--output-format", "stream-json", + "--dangerously-skip-permissions", "--print-timeout", "180s", "--add-dir", str(sb.cwd), *extra] + try: + res = print_ask(argv, sb.cwd, sb.env, sb.root / "agy.stderr.log", self.turn_timeout, extract=extract) + finally: + self.cleanup_hook(sb) + if named: + res.notes = (res.notes + f" slash=/{named.group(0)}").strip() + return res + + def cleanup_hook(self, sb: Sandbox) -> None: + return None + + +def slash_prefixed(prompt: str) -> str: + """agy exposes skills as slash commands that expand in place: invoking the listed skill by + name is its native load, so the prompt leads with it when it names one.""" + named = NAME_RE.search(prompt) + return f"/{named.group(0)} {prompt}" if named else prompt + + +class _SlashPromptSession(Session): + def __init__(self, inner: Session, cleanup) -> None: + self.inner = inner + self.cleanup = cleanup + + def ask(self, prompt: str) -> AskResult: + return self.inner.ask(slash_prefixed(prompt)) + + def reload(self) -> str | None: + return self.inner.reload() + + def close(self) -> None: + try: + self.inner.close() + finally: + self.cleanup() + + +class AgyDirLayoutAdapter(AgyAdapter): + entry = "agy-dirlayout" + flat_skill_layout = False + + +class AgyCliDirAdapter(AgyAdapter): + """The CLI documentation's global plugin directory: written into the real home for the turn + and removed right after, since no private home is available.""" + + entry = "agy-clidir" + flat_skill_layout = False + + def hook_dir(self, sb: Sandbox) -> Path: + return Path.home() / ".gemini" / "antigravity-cli" / "plugins" / "kcap-probe" + + def install_startup_hook(self, sb: Sandbox, script: Path) -> HookInfo: + info = super().install_startup_hook(sb, script) + return HookInfo(mechanism="~/.gemini/antigravity-cli/plugins PreInvocation", config_path=info.config_path) + + def cleanup_hook(self, sb: Sandbox) -> None: + shutil.rmtree(self.hook_dir(sb), ignore_errors=True) diff --git a/docs/probes/2026-09-16-skills-discovery/harness/base.py b/docs/probes/2026-09-16-skills-discovery/harness/base.py new file mode 100644 index 0000000000..c20ff2b0b9 --- /dev/null +++ b/docs/probes/2026-09-16-skills-discovery/harness/base.py @@ -0,0 +1,154 @@ +from __future__ import annotations + +import shutil +import subprocess +from collections.abc import Callable, Mapping +from dataclasses import dataclass +from pathlib import Path +from types import MappingProxyType + +from lib.isolation import Sandbox + + +@dataclass +class HookInfo: + mechanism: str + config_path: str + # Where this mechanism actually writes the skill, when that is not the arm's own path. + target: Path | None = None + + +@dataclass +class AskResult: + reply_text: str + raw: str + argv: list[str] + started_at: float + first_request_at: float + stderr_path: str | None + exit_code: int | None + notes: str = "" + + +class Session: + """A live vendor session that takes more than one prompt.""" + + def ask(self, prompt: str) -> AskResult: + raise NotImplementedError + + def reload(self) -> str | None: + """Ask the vendor to rebuild its skill catalogue: the command used, or None if it has none.""" + return None + + def close(self) -> None: + return None + + +class ClassifiedSession(Session): + """A driver session whose replies carry the adapter's own tool classification instead of the + driver's flat count, which cannot tell a listed skill's own file from a search for it.""" + + def __init__(self, inner: Session, classify: Callable[[AskResult], str]) -> None: + self.inner = inner + self.classify = classify + + def ask(self, prompt: str) -> AskResult: + res = self.inner.ask(prompt) + generic = " ".join(n for n in res.notes.split() if not n.startswith("tools_used=")) + res.notes = (generic + " " + self.classify(res)).strip() + return res + + def reload(self) -> str | None: + return self.inner.reload() + + def close(self) -> None: + self.inner.close() + + +class Adapter: + entry: str = "" + harness: str = "" + binary: str = "" + lever: str = "" + # Class-level defaults are shared by every adapter, so they are immutable: a subclass that + # appended to a list default would extend it for all of them. + credential_files: tuple[str, ...] = () + passthrough_env: tuple[str, ...] = () + extra_env: Mapping[str, str] = MappingProxyType({}) + native_root: str = "" + documented_roots: frozenset[str] = frozenset() + flat_skill_layout: bool = False + modes: tuple[str, ...] = ("print", "daemon") + turn_timeout: float = 180.0 + can_resume: bool = False + # Interactive launch: (regex on the stripped screen, keys to send) pairs for the vendor's + # dialogs, the slash command that rebuilds its catalogue, the keys that end it. + tui_dialogs: tuple[tuple[str, str], ...] = () + tui_reload: str | None = None + # An interactive turn carries the UI's own latency on top of the model's, so it gets longer + # than a headless one before the screen is called unreadable. + tui_timeout: float = 300.0 + tui_exit: tuple[str, ...] = ("\x03", "\x03", "\x04") + tui_ready: float = 4.0 + + def binary_path(self) -> str | None: + return shutil.which(self.binary) + + def version(self, env: dict | None = None) -> str: + path = self.binary_path() + if path is None: + return "not-installed" + kwargs = {"env": env} if env is not None else {} + out = subprocess.run([path, "--version"], capture_output=True, text=True, timeout=60, **kwargs) + return (out.stdout or out.stderr).strip().splitlines()[0] if (out.stdout or out.stderr).strip() else "unknown" + + def real_root(self) -> Path | None: + return None + + def prepare(self, sb: Sandbox) -> None: + return None + + def check_auth(self, sb: Sandbox) -> bool | None: + return None + + def install_startup_hook(self, sb: Sandbox, script: Path) -> HookInfo | None: + return None + + def install_registration(self, sb: Sandbox, skill_file: Path, body: str) -> HookInfo | None: + return None + + def ask(self, sb: Sandbox, mode: str, prompt: str) -> AskResult: + raise NotImplementedError + + def open_session(self, sb: Sandbox, mode: str) -> Session | None: + if mode != "tui": + return None + argv = self.tui_argv(sb) + if argv is None: + return None + from lib.pty_driver import PtySession + session = PtySession(argv, sb.cwd, sb.env, sb.root / f"{self.harness}-tui.log", dialogs=self.tui_dialogs, + reload_command=self.tui_reload, exit_keys=self.tui_exit, ready_idle=self.tui_ready, + timeout=self.tui_timeout) + session.start() + return session + + def tui_argv(self, sb: Sandbox) -> list[str] | None: + return None + + def session_id(self, res: AskResult) -> str | None: + return None + + def resume(self, sb: Sandbox, session_id: str, prompt: str) -> AskResult | None: + return None + + def list_catalogue(self, sb: Sandbox) -> str | None: + return None + + def skill_dir(self, sb: Sandbox, root: str, name: str) -> Path: + return sb.repo / root if self.flat_skill_layout else sb.repo / root / name + + def skill_file(self, sb: Sandbox, root: str, name: str) -> Path: + if self.flat_skill_layout: + return sb.repo / root / f"{name}.md" + return sb.repo / root / name / "SKILL.md" diff --git a/docs/probes/2026-09-16-skills-discovery/harness/claude.py b/docs/probes/2026-09-16-skills-discovery/harness/claude.py new file mode 100644 index 0000000000..ed51fec69c --- /dev/null +++ b/docs/probes/2026-09-16-skills-discovery/harness/claude.py @@ -0,0 +1,122 @@ +from __future__ import annotations + +import json +import os +import subprocess +import tempfile +from pathlib import Path + +from harness.base import Adapter, AskResult, HookInfo +from lib.isolation import Sandbox +from lib.print_driver import print_ask + + +class ClaudeAdapter(Adapter): + entry = "claude" + harness = "claude" + binary = "claude" + lever = "CLAUDE_CONFIG_DIR" + credential_files = (".credentials.json",) + native_root = ".claude/skills" + documented_roots = frozenset({".claude/skills"}) + modes = ("print", "tui") + can_resume = True + # An arrow and its Enter are typed a redraw apart; the browser dialog's default keeps the + # browser tools off, which is what the probe wants. + tui_dialogs = ((r"Yes, I trust this folder", "\x1b[B\r"), (r"keep browser tools off", "\r"), + (r"Teach auto mode about your environment", "\x1b")) + tui_exit = ("/exit\r", "\x03", "\x03") + tui_ready = 5.0 + + def __init__(self) -> None: + forced = os.environ.get("KCAP_PROBE_CLAUDE_REAL_CONFIG") + self.real_config = forced == "1" if forced in ("0", "1") else not self._isolated_login_works() + + def real_root(self) -> Path | None: + return Path(os.environ.get("CLAUDE_CONFIG_DIR") or (Path.home() / ".claude")) + + def _auth_status(self, env: dict) -> bool: + binary = self.binary_path() + if binary is None: + return False + try: + out = subprocess.run([binary, "auth", "status"], env=env, capture_output=True, text=True, timeout=60) + return bool(json.loads(out.stdout).get("loggedIn")) + except (OSError, subprocess.SubprocessError, json.JSONDecodeError, AttributeError): + return False + + def _isolated_login_works(self) -> bool: + """A keychain-held login is keyed by the config root it was minted under, so a private + root can report logged out; that decides the fallback once per adapter instance.""" + if self.binary_path() is None: + return True + with tempfile.TemporaryDirectory(prefix="skprobe-claude-") as d: + env = {k: v for k, v in os.environ.items() if k in ("PATH", "HOME", "TERM", "USER", "LOGNAME")} + env["CLAUDE_CONFIG_DIR"] = d + return self._auth_status(env) + + def prepare(self, sb: Sandbox) -> None: + settings = sb.config_root / "settings.json" + if not settings.exists(): + settings.write_text("{}\n") + if self.real_config: + # The keychain entry is keyed by the config root, and naming the default root + # explicitly counts as a different one: the variable must be absent, not defaulted. + # The hooks still come from the sandbox's settings file, passed with --settings; + # KCAP_SKIP stands kcap's own hooks down should any user-level one still load. + sb.env.pop("CLAUDE_CONFIG_DIR", None) + sb.env["KCAP_SKIP"] = "1" + return + (sb.config_root / ".claude.json").write_text(json.dumps({ + "hasCompletedOnboarding": True, + "projects": {str(sb.repo): {"hasTrustDialogAccepted": True}}, + }, indent=2) + "\n") + + def check_auth(self, sb: Sandbox) -> bool | None: + return self._auth_status(sb.env) + + def install_startup_hook(self, sb: Sandbox, script: Path) -> HookInfo: + settings = sb.config_root / "settings.json" + data = json.loads(settings.read_text() or "{}") + data.setdefault("hooks", {})["SessionStart"] = [ + {"hooks": [{"type": "command", "command": str(script), "timeout": 10}]} + ] + settings.write_text(json.dumps(data, indent=2) + "\n") + where = "--settings file over the real config root" if self.real_config else "isolated settings.json" + return HookInfo(mechanism=f"{where} hooks.SessionStart", config_path=str(settings)) + + def _settings_argv(self, sb: Sandbox) -> list[str]: + if self.real_config: + return ["--setting-sources", "project", "--settings", str(sb.config_root / "settings.json")] + return ["--setting-sources", "user"] + + def print_argv(self, sb: Sandbox, prompt: str) -> list[str]: + return [self.binary_path() or self.binary, "-p", prompt, "--output-format", "json", "--max-turns", "4", + "--strict-mcp-config", "--allowedTools", "Skill"] + self._settings_argv(sb) + + def tui_argv(self, sb: Sandbox) -> list[str] | None: + return [self.binary_path() or self.binary, "--strict-mcp-config", + "--allowedTools", "Skill"] + self._settings_argv(sb) + + def session_id(self, res: AskResult) -> str | None: + try: + return json.loads(res.raw).get("session_id") + except (json.JSONDecodeError, AttributeError): + return None + + def resume(self, sb: Sandbox, session_id: str, prompt: str) -> AskResult | None: + return self._run(sb, self.print_argv(sb, prompt) + ["--resume", session_id]) + + def _run(self, sb: Sandbox, argv: list[str]) -> AskResult: + def extract(raw: str) -> str: + obj = json.loads(raw) + if obj.get("is_error"): + raise ValueError(f"is_error: {json.dumps(obj)[:300]}") + return obj.get("result") or "" + + res = print_ask(argv, sb.cwd, sb.env, sb.root / "claude.stderr.log", self.turn_timeout, extract=extract) + res.notes = (res.notes + f" config={'real' if self.real_config else 'isolated'}").strip() + return res + + def ask(self, sb: Sandbox, mode: str, prompt: str) -> AskResult: + return self._run(sb, self.print_argv(sb, prompt)) diff --git a/docs/probes/2026-09-16-skills-discovery/harness/codex.py b/docs/probes/2026-09-16-skills-discovery/harness/codex.py new file mode 100644 index 0000000000..0cda613d56 --- /dev/null +++ b/docs/probes/2026-09-16-skills-discovery/harness/codex.py @@ -0,0 +1,159 @@ +from __future__ import annotations + +import json +import subprocess +from pathlib import Path + +from harness.base import Adapter, AskResult, ClassifiedSession, HookInfo, Session +from lib.appserver_driver import AppServerSession, appserver_ask +from lib.isolation import Sandbox +from lib.print_driver import print_ask + +class CodexAdapter(Adapter): + entry = "codex" + harness = "codex" + binary = "codex" + lever = "CODEX_HOME" + credential_files = ("auth.json",) + native_root = ".agents/skills" + documented_roots = frozenset({".agents/skills", ".codex/skills"}) + modes = ("print", "daemon", "tui") + can_resume = True + # A linear transcript is easier to read back than an alternate-screen redraw. + tui_exit = ("\x03", "\x03", "\x04") + + def real_root(self) -> Path | None: + return Path.home() / ".codex" + + def prepare(self, sb: Sandbox) -> None: + (sb.config_root / "config.toml").write_text( + f'[projects.{json.dumps(str(sb.repo))}]\ntrust_level = "trusted"\n') + + def check_auth(self, sb: Sandbox) -> bool | None: + out = subprocess.run([self.binary_path() or self.binary, "login", "status"], env=sb.env, + capture_output=True, text=True, timeout=60) + return out.returncode == 0 and "Logged in" in (out.stdout + out.stderr) + + def install_startup_hook(self, sb: Sandbox, script: Path) -> HookInfo: + hooks = sb.config_root / "hooks.json" + hooks.write_text(json.dumps({"hooks": {"SessionStart": [ + {"hooks": [{"type": "command", "command": str(script), "timeout": 30}]} + ]}}, indent=2) + "\n") + return HookInfo(mechanism="hooks.json SessionStart", config_path=str(hooks)) + + def tui_argv(self, sb: Sandbox) -> list[str] | None: + # The same hook-trust bypass the headless modes pass: without it an unsigned hook is + # silently not run, which reads as a harness that has no startup hook at all. + return [self.binary_path() or self.binary, "--sandbox", "read-only", "-a", "never", + "--dangerously-bypass-hook-trust", "--no-alt-screen"] + + def open_session(self, sb: Sandbox, mode: str) -> Session | None: + if mode != "daemon": + return super().open_session(sb, mode) + inner = AppServerSession(self.binary_path() or self.binary, sb.cwd, sb.env, + sb.root / "codex-appserver.stderr.log", self.turn_timeout) + inner.start() + return ClassifiedSession(inner, lambda r: classify_tool_items(appserver_items(r.raw))) + + def session_id(self, res: AskResult) -> str | None: + for line in res.raw.splitlines(): + try: + obj = json.loads(line) + except json.JSONDecodeError: + continue + if obj.get("thread_id"): + return obj["thread_id"] + return None + + def resume(self, sb: Sandbox, session_id: str, prompt: str) -> AskResult | None: + return self._exec(sb, prompt, resume=session_id) + + def ask(self, sb: Sandbox, mode: str, prompt: str) -> AskResult: + binary = self.binary_path() or self.binary + if mode == "daemon": + res = appserver_ask(binary, sb.cwd, sb.env, prompt, sb.root / "codex-appserver.stderr.log", + self.turn_timeout) + # The driver's flat tool count is replaced by the classification that knows a listed read. + generic = " ".join(n for n in res.notes.split() if not n.startswith("tools_used=")) + res.notes = (generic + " " + classify_tool_items(appserver_items(res.raw))).strip() + return res + return self._exec(sb, prompt) + + def _exec(self, sb: Sandbox, prompt: str, resume: str | None = None) -> AskResult: + binary = self.binary_path() or self.binary + last = sb.root / "last-message.txt" + # A file left by an earlier turn would be read as this turn's answer. + last.unlink(missing_ok=True) + # `exec resume` takes a narrower flag set than `exec`: the sandbox and colour flags it + # rejects outright, and the session's own recorded sandbox applies instead. + head = [binary, "exec", *(["resume", resume] if resume else [])] + sandbox = [] if resume else ["--sandbox", "read-only", "--color", "never"] + argv = head + ["--json", "--skip-git-repo-check", *sandbox, + "--dangerously-bypass-hook-trust", "--output-last-message", str(last), "-"] + + def extract(raw: str) -> str: + if last.exists(): + return last.read_text().strip() + texts = [] + for line in raw.splitlines(): + try: + obj = json.loads(line) + except json.JSONDecodeError: + continue + item = obj.get("item") or {} + if obj.get("type") == "item.completed" and item.get("type") == "agent_message": + texts.append(item.get("text", "")) + return "\n".join(texts) + + res = print_ask(argv, sb.cwd, sb.env, sb.root / "codex.stderr.log", self.turn_timeout, + stdin_text=prompt, extract=extract) + res.notes = (res.notes + " " + classify_tool_items(exec_items(res.raw))).strip() + return res + + +SEARCH_MARKERS = ("find ", "grep", "rg ", "ls ", "locate ", "*", "tree ") + + +def exec_items(raw: str) -> list[tuple[str, str]]: + """(item type, command) for every completed item in `codex exec --json` output.""" + items = [] + for line in raw.splitlines(): + try: + obj = json.loads(line) + except json.JSONDecodeError: + continue + item = obj.get("item") or {} + if obj.get("type") == "item.completed": + items.append((item.get("type") or "", item.get("command") or "")) + return items + + +def appserver_items(raw: str) -> list[tuple[str, str]]: + """(item type, command) for every item/completed notification in an app-server frame log.""" + items = [] + try: + frames = json.loads(raw) + except json.JSONDecodeError: + return items + for f in frames: + fr = f.get("frame") or {} + if fr.get("method") == "item/completed": + item = (fr.get("params") or {}).get("item") or {} + items.append((item.get("type") or "", item.get("command") or "")) + return items + + +def classify_tool_items(items: list[tuple[str, str]]) -> str: + """Codex loads a listed skill by reading its file, so a `cat` of a SKILL.md path is the native + mechanism at work; a search command is what would make a sighting inconclusive.""" + skill_reads = searches = other = 0 + for kind, command in items: + if kind in ("agent_message", "agentMessage", "reasoning", "userMessage", "user_message", "error", ""): + continue + if any(m in command for m in SEARCH_MARKERS): + searches += 1 + elif "SKILL.md" in command: + skill_reads += 1 + else: + other += 1 + return f"tools_used={searches + other} skill_reads={skill_reads} searches={searches}" diff --git a/docs/probes/2026-09-16-skills-discovery/harness/copilot.py b/docs/probes/2026-09-16-skills-discovery/harness/copilot.py new file mode 100644 index 0000000000..7398571db6 --- /dev/null +++ b/docs/probes/2026-09-16-skills-discovery/harness/copilot.py @@ -0,0 +1,161 @@ +from __future__ import annotations + +import json +import subprocess +from pathlib import Path + +from harness.base import Adapter, AskResult, ClassifiedSession, HookInfo, Session +from lib.acp_driver import AcpSession, acp_ask +from lib.isolation import Sandbox +from lib.print_driver import print_ask + + +class CopilotAdapter(Adapter): + entry = "copilot" + harness = "copilot" + binary = "copilot" + lever = "COPILOT_HOME" + native_root = ".github/skills" + documented_roots = frozenset({".github/skills", ".agents/skills", ".claude/skills"}) + modes = ("print", "daemon", "tui") + can_resume = True + tui_dialogs = ((r"Do you trust the files", "\r"),) + tui_reload = "/skills reload" + tui_exit = ("/exit\r", "\x03", "\x03") + + def __init__(self) -> None: + try: + out = subprocess.run(["gh", "auth", "token"], capture_output=True, text=True, timeout=30) + self.token = out.stdout.strip() if out.returncode == 0 else "" + except (OSError, subprocess.SubprocessError): + self.token = "" + + def real_root(self) -> Path | None: + return Path.home() / ".copilot" + + def prepare(self, sb: Sandbox) -> None: + if self.token: + sb.env["COPILOT_GITHUB_TOKEN"] = self.token + + def check_auth(self, sb: Sandbox) -> bool | None: + return bool(self.token) + + def install_startup_hook(self, sb: Sandbox, script: Path) -> HookInfo: + hooks = sb.config_root / "hooks" + hooks.mkdir(parents=True, exist_ok=True) + path = hooks / "probe.json" + path.write_text(json.dumps({"version": 1, "hooks": {"sessionStart": [ + {"type": "command", "command": str(script), "timeoutSec": 30} + ]}}, indent=2) + "\n") + return HookInfo(mechanism="hooks/*.json sessionStart", config_path=str(path)) + + def acp_argv(self) -> list[str]: + return [self.binary_path() or self.binary, "--acp", "--stdio"] + + def tui_argv(self, sb: Sandbox) -> list[str] | None: + return [self.binary_path() or self.binary, "--allow-all-tools"] + + def open_session(self, sb: Sandbox, mode: str) -> Session | None: + if mode != "daemon": + return super().open_session(sb, mode) + inner = AcpSession(self.acp_argv(), sb.cwd, sb.env, sb.root / "copilot-acp.stderr.log", self.turn_timeout) + inner.start() + return ClassifiedSession(inner, lambda r: classify_copilot_tools(acp_tool_calls(r.raw))) + + def session_id(self, res: AskResult) -> str | None: + for line in res.raw.splitlines(): + try: + obj = json.loads(line) + except json.JSONDecodeError: + continue + if obj.get("type") == "result" and obj.get("sessionId"): + return obj["sessionId"] + return None + + def resume(self, sb: Sandbox, session_id: str, prompt: str) -> AskResult | None: + return self._print(sb, prompt, [f"--resume={session_id}"]) + + def list_catalogue(self, sb: Sandbox) -> str | None: + out = subprocess.run([self.binary_path() or self.binary, "skill", "list"], cwd=str(sb.repo), + env=sb.env, capture_output=True, text=True, timeout=120) + return out.stdout + + def ask(self, sb: Sandbox, mode: str, prompt: str) -> AskResult: + if mode == "daemon": + res = acp_ask(self.acp_argv(), sb.cwd, sb.env, prompt, sb.root / "copilot-acp.stderr.log", + self.turn_timeout) + generic = " ".join(n for n in res.notes.split() if not n.startswith("tools_used=")) + res.notes = (generic + " " + classify_copilot_tools(acp_tool_calls(res.raw))).strip() + return res + return self._print(sb, prompt) + + def _print(self, sb: Sandbox, prompt: str, extra: list[str] = ()) -> AskResult: + binary = self.binary_path() or self.binary + argv = [binary, "-p", prompt, "--allow-all-tools", "--output-format", "json", *extra] + + def extract(raw: str) -> str: + texts = [] + for line in raw.splitlines(): + try: + obj = json.loads(line) + except json.JSONDecodeError: + continue + if obj.get("type") == "assistant.message": + content = (obj.get("data") or {}).get("content") + if isinstance(content, str) and content.strip(): + texts.append(content) + return "\n".join(texts) + + res = print_ask(argv, sb.cwd, sb.env, sb.root / "copilot.stderr.log", self.turn_timeout, extract=extract) + res.notes = (res.notes + " " + classify_copilot_tools(print_tool_calls(res.raw))).strip() + return res + + +SEARCH_TOOLS = ("bash", "shell", "grep", "glob", "find", "view", "read", "ls") + + +def print_tool_calls(raw: str) -> list[tuple[str, str]]: + """(tool name, title) for every tool.execution_start event in `copilot --output-format json`.""" + calls = [] + for line in raw.splitlines(): + try: + obj = json.loads(line) + except json.JSONDecodeError: + continue + if obj.get("type") == "tool.execution_start": + data = obj.get("data") or {} + calls.append((data.get("toolName") or "", json.dumps(data.get("arguments") or {}))) + return calls + + +def acp_tool_calls(raw: str) -> list[tuple[str, str]]: + """(kind, title) for every tool_call update in an ACP frame log.""" + calls = [] + try: + frames = json.loads(raw) + except json.JSONDecodeError: + return calls + for f in frames: + fr = f.get("frame") or {} + if fr.get("method") != "session/update": + continue + upd = (fr.get("params") or {}).get("update") or {} + if upd.get("sessionUpdate") == "tool_call": + title = upd.get("title") or "" + name = "skill" if title.startswith("Using skill:") else (upd.get("kind") or "") + calls.append((name, title)) + return calls + + +def classify_copilot_tools(calls: list[tuple[str, str]]) -> str: + """Copilot loads a listed skill through its own `skill` tool; any other tool is either a + search for the file or unrelated work, and both make a sighting inconclusive.""" + skill_loads = searches = other = 0 + for name, _ in calls: + if name == "skill": + skill_loads += 1 + elif name in SEARCH_TOOLS: + searches += 1 + else: + other += 1 + return f"tools_used={searches + other} skill_loads={skill_loads} searches={searches}" diff --git a/docs/probes/2026-09-16-skills-discovery/harness/cursor.py b/docs/probes/2026-09-16-skills-discovery/harness/cursor.py new file mode 100644 index 0000000000..828e5bba47 --- /dev/null +++ b/docs/probes/2026-09-16-skills-discovery/harness/cursor.py @@ -0,0 +1,240 @@ +from __future__ import annotations + +import json +import os +import subprocess +from pathlib import Path + +from harness.base import Adapter, AskResult, ClassifiedSession, HookInfo, Session +from lib.acp_driver import AcpSession, acp_ask +from lib.isolation import Sandbox +from lib.print_driver import print_ask + + +class CursorAdapter(Adapter): + """Cursor CLI. Its login is keyring-held and not found from a private HOME, so the CLI runs + against the real home with kcap's hooks stood down; the probe hook is the sandbox repository's + own `.cursor/hooks.json`, which Cursor reads beside the user-level one.""" + + entry = "cursor" + harness = "cursor" + binary = "cursor-agent" + lever = "CURSOR_PROBE_UNUSED" + native_root = ".cursor/skills" + documented_roots = frozenset({".cursor/skills", ".agents/skills", ".claude/skills", ".codex/skills"}) + modes = ("print", "daemon", "tui") + can_resume = True + tui_exit = ("/exit\r", "\x03", "\x03") + + def real_root(self) -> Path | None: + return Path.home() + + def prepare(self, sb: Sandbox) -> None: + sb.env.pop(self.lever, None) + sb.env["HOME"] = os.environ.get("HOME", str(Path.home())) + sb.env["KCAP_SKIP"] = "1" + + def check_auth(self, sb: Sandbox) -> bool | None: + out = subprocess.run([self.binary_path() or self.binary, "status"], env=sb.env, capture_output=True, + text=True, timeout=60) + return out.returncode == 0 and "Logged in" in (out.stdout + out.stderr) + + def _hooks(self, sb: Sandbox) -> Path: + d = sb.repo / ".cursor" + d.mkdir(parents=True, exist_ok=True) + return d / "hooks.json" + + def install_startup_hook(self, sb: Sandbox, script: Path) -> HookInfo: + self._hooks(sb).write_text(json.dumps({"version": 1, "hooks": {"sessionStart": [{"command": str(script)}]}}, + indent=2) + "\n") + return HookInfo(mechanism="project .cursor/hooks.json sessionStart", config_path=str(self._hooks(sb))) + + def install_registration(self, sb: Sandbox, skill_file: Path, body: str) -> HookInfo | None: + plugin = sb.root / "plugin" + (plugin / ".cursor-plugin").mkdir(parents=True, exist_ok=True) + (plugin / ".cursor-plugin" / "plugin.json").write_text(json.dumps({"name": "kcap-probe", "version": "1.0.0"})) + target = plugin / "skills" / skill_file.parent.name / "SKILL.md" + if body and not body.endswith("\n"): + body += "\n" + script = sb.config_root / "probe-workspace-open.sh" + script.write_text( + "#!/bin/sh\nset -eu\n" + f"mkdir -p '{target.parent}'\n" + f"cat > '{target}' <<'KCAP_PROBE_EOF'\n{body}KCAP_PROBE_EOF\n" + f"printf '{{\"fired_at\": %s}}\\n' \"$(date +%s)\" > '{sb.config_root / 'probe-hook-fired.json'}'\n" + f"printf '%s\\n' '{json.dumps({'pluginPaths': [str(plugin)]})}'\n") + script.chmod(0o755) + self._hooks(sb).write_text(json.dumps({"version": 1, "hooks": {"workspaceOpen": [{"command": str(script)}]}}, + indent=2) + "\n") + return HookInfo(mechanism="project .cursor/hooks.json workspaceOpen pluginPaths", + config_path=str(self._hooks(sb)), target=target) + + def acp_argv(self) -> list[str]: + return [self.binary_path() or self.binary, "acp", "--trust"] + + def tui_argv(self, sb: Sandbox) -> list[str] | None: + return [self.binary_path() or self.binary, "--trust"] + + def open_session(self, sb: Sandbox, mode: str) -> Session | None: + if mode != "daemon": + return super().open_session(sb, mode) + inner = AcpSession(self.acp_argv(), sb.cwd, sb.env, sb.root / "cursor-acp.stderr.log", self.turn_timeout) + inner.start() + return ClassifiedSession(inner, lambda r: classify_cursor_tools(r.raw)) + + def session_id(self, res: AskResult) -> str | None: + for line in res.raw.splitlines(): + try: + obj = json.loads(line) + except json.JSONDecodeError: + continue + if isinstance(obj.get("session_id"), str): + return obj["session_id"] + return None + + def resume(self, sb: Sandbox, session_id: str, prompt: str) -> AskResult | None: + return self._print(sb, prompt, [f"--resume={session_id}"]) + + def ask(self, sb: Sandbox, mode: str, prompt: str) -> AskResult: + if mode == "daemon": + res = acp_ask(self.acp_argv(), sb.cwd, sb.env, prompt, sb.root / "cursor-acp.stderr.log", + self.turn_timeout) + generic = " ".join(n for n in res.notes.split() if not n.startswith("tools_used=")) + res.notes = (generic + " " + classify_cursor_tools(res.raw)).strip() + return res + return self._print(sb, prompt) + + def _print(self, sb: Sandbox, prompt: str, extra: list[str] = ()) -> AskResult: + binary = self.binary_path() or self.binary + + def extract(raw: str) -> str: + texts = [] + for line in raw.splitlines(): + try: + obj = json.loads(line) + except json.JSONDecodeError: + continue + if obj.get("type") == "assistant": + for part in (obj.get("message") or {}).get("content") or []: + if isinstance(part, dict) and isinstance(part.get("text"), str): + texts.append(part["text"]) + elif obj.get("type") == "result" and isinstance(obj.get("result"), str): + texts.append(obj["result"]) + return "\n".join(texts) + + # Print mode ends every tool-using turn with "WritableIterable is closed" and exit 1; the + # json and text formats then print nothing, while stream-json has already streamed the + # answer, so the answer is read from the stream and the exit code is recorded beside it. + argv = [binary, "-p", "--output-format", "stream-json", "--trust", "--force", *extra, prompt] + res = print_ask(argv, sb.cwd, sb.env, sb.root / "cursor.stderr.log", self.turn_timeout, extract=extract) + res.notes = (res.notes + " " + classify_cursor_stream(res.raw)).strip() + return res + + +class CursorUserHooksAdapter(CursorAdapter): + """The user-level `~/.cursor/hooks.json`, where kcap installs its own hooks: the probe entry is + merged into the real file for one turn and the original file is restored afterwards.""" + + entry = "cursor-userhooks" + + def __init__(self) -> None: + self._backup: bytes | None = None + self._touched = False + + def _user_hooks(self) -> Path: + return Path.home() / ".cursor" / "hooks.json" + + def _merge(self, sb: Sandbox, event: str, command: str) -> HookInfo: + path = self._user_hooks() + path.parent.mkdir(parents=True, exist_ok=True) + self._backup = path.read_bytes() if path.exists() else None + self._touched = True + try: + data = json.loads(self._backup.decode()) if self._backup else {} + except json.JSONDecodeError: + data = {} + data.setdefault("version", 1) + data.setdefault("hooks", {}).setdefault(event, []).append({"command": command}) + path.write_text(json.dumps(data, indent=2) + "\n") + return HookInfo(mechanism=f"user ~/.cursor/hooks.json {event}", config_path=str(path)) + + def install_startup_hook(self, sb: Sandbox, script: Path) -> HookInfo: + return self._merge(sb, "sessionStart", str(script)) + + def install_registration(self, sb: Sandbox, skill_file: Path, body: str) -> HookInfo | None: + info = super().install_registration(sb, skill_file, body) + script = json.loads(Path(info.config_path).read_text())["hooks"]["workspaceOpen"][0]["command"] + Path(info.config_path).unlink() + merged = self._merge(sb, "workspaceOpen", script) + # The plugin directory the hook writes into is the same one either file points at. + merged.target = info.target + return merged + + def cleanup_hook(self, sb: Sandbox) -> None: + if not self._touched: + return + path = self._user_hooks() + if self._backup is None: + path.unlink(missing_ok=True) + else: + path.write_bytes(self._backup) + self._touched = False + + def ask(self, sb: Sandbox, mode: str, prompt: str) -> AskResult: + try: + return super().ask(sb, mode, prompt) + finally: + self.cleanup_hook(sb) + + +def classify_cursor_stream(raw: str) -> str: + """The stream-json events of print mode: a tool_call whose input names a SKILL.md path is the + native load; any shell or search tool call makes a sighting inconclusive.""" + skill_reads = searches = other = 0 + for line in raw.splitlines(): + try: + obj = json.loads(line) + except json.JSONDecodeError: + continue + if obj.get("type") != "tool_call" or obj.get("subtype") not in (None, "started"): + continue + call = json.dumps(obj.get("tool_call") or obj) + if "SKILL.md" in call and ("read" in call.lower()): + skill_reads += 1 + elif any(k in call for k in ("shellToolCall", "grep", "glob", "find", "ls")): + searches += 1 + else: + other += 1 + return f"tools_used={searches + other} skill_reads={skill_reads} searches={searches}" + + +def classify_cursor_tools(raw: str) -> str: + """Cursor lists a skill with its path and the model reads that file, so a read of a SKILL.md + path is the native mechanism; a shell or search tool is what makes a sighting inconclusive.""" + calls: dict[str, dict] = {} + try: + frames = json.loads(raw) + except json.JSONDecodeError: + frames = [] + for f in frames: + fr = f.get("frame") or {} + if fr.get("method") != "session/update": + continue + upd = (fr.get("params") or {}).get("update") or {} + if not upd.get("sessionUpdate", "").startswith("tool_call"): + continue + call = calls.setdefault(upd.get("toolCallId") or "", {"kind": "", "path": ""}) + if upd.get("kind"): + call["kind"] = upd["kind"] + path = (upd.get("rawInput") or {}).get("path") + if isinstance(path, str): + call["path"] = path + skill_reads = searches = other = 0 + for call in calls.values(): + if call["kind"] == "read" and call["path"].endswith("SKILL.md"): + skill_reads += 1 + elif call["kind"] in ("execute", "search", "fetch"): + searches += 1 + else: + other += 1 + return f"tools_used={searches + other} skill_reads={skill_reads} searches={searches}" diff --git a/docs/probes/2026-09-16-skills-discovery/harness/fake.py b/docs/probes/2026-09-16-skills-discovery/harness/fake.py new file mode 100644 index 0000000000..2deddcd4cf --- /dev/null +++ b/docs/probes/2026-09-16-skills-discovery/harness/fake.py @@ -0,0 +1,100 @@ +from __future__ import annotations + +import subprocess +import sys +import time +from pathlib import Path + +from harness.base import Adapter, AskResult, HookInfo, Session +from lib.isolation import Sandbox +from lib.probe_skill import NO_SKILL, TOKEN_RE + + +class FakeSession(Session): + def __init__(self, adapter: "FakeAdapter", sb: Sandbox) -> None: + self.adapter = adapter + self.sb = sb + # A frozen catalogue is what a vendor that indexes once looks like; /reload refreshes it. + self.frozen: list[str] | None = None if adapter.live_catalogue else adapter.catalogue(sb) + + def ask(self, prompt: str) -> AskResult: + lines = self.frozen if self.frozen is not None else self.adapter.catalogue(self.sb) + return self.adapter.reply(lines, ["fake", "--session"]) + + def reload(self) -> str | None: + if self.frozen is not None: + self.frozen = self.adapter.catalogue(self.sb) + return "/reload" + + +class FakeAdapter(Adapter): + entry = "fake" + harness = "fake" + binary = "sh" + lever = "FAKE_HOME" + native_root = ".fake/skills" + documented_roots = frozenset({".fake/skills", ".agents/skills"}) + modes = ("print", "daemon", "tui") + can_resume = True + passthrough_env = ("KCAP_FAKE_TUI_BARE",) + # A screen the fake never writes to is waited on for this long, so the suite stays quick. + turn_timeout = 15.0 + tui_timeout = 15.0 + tui_reload = "/reload" + tui_ready = 0.3 + # What the fake vendor actually loads, which a subclass keeps while narrowing what it documents. + read_roots = (".agents/skills", ".fake/skills") + live_catalogue = True + + def __init__(self) -> None: + # Keyed by sandbox root: a real vendor's hook lives in that sandbox's own config, and one + # adapter instance now runs every scenario's arms, each in its own sandbox in turn. + self._hooks: dict[Path, Path] = {} + + def version(self, env: dict | None = None) -> str: + return "1.0" + + def install_startup_hook(self, sb: Sandbox, script: Path) -> HookInfo: + self._hooks[sb.root] = script + return HookInfo(mechanism="fake-startup", config_path=str(script)) + + def catalogue(self, sb: Sandbox) -> list[str]: + lines = [] + for root in self.read_roots: + for skill_md in sorted((sb.cwd / root).glob("kcap-probe-*/SKILL.md")): + m = TOKEN_RE.search(skill_md.read_text()) + if m: + lines.append(f"{skill_md.parent.name}=PROBE-BODY-{m.group(1)}") + return lines + + def reply(self, lines: list[str], argv: list[str]) -> AskResult: + started = time.time() + text = "\n".join(lines) if lines else NO_SKILL + return AskResult(reply_text=text, raw=text, argv=argv, started_at=started, + first_request_at=started, stderr_path=None, exit_code=0) + + def ask(self, sb: Sandbox, mode: str, prompt: str) -> AskResult: + hook = self._hooks.get(sb.root) + if hook is not None: + subprocess.run([str(hook)], input="{}", capture_output=True, text=True, timeout=10) + return self.reply(self.catalogue(sb), ["fake"]) + + def open_session(self, sb: Sandbox, mode: str) -> Session | None: + if mode == "daemon": + return FakeSession(self, sb) + return super().open_session(sb, mode) + + def session_id(self, res: AskResult) -> str | None: + return "fake-session" + + def resume(self, sb: Sandbox, session_id: str, prompt: str) -> AskResult | None: + return self.reply(self.catalogue(sb), ["fake", "--resume", session_id]) + + def tui_argv(self, sb: Sandbox) -> list[str] | None: + servers = Path(__file__).resolve().parent.parent / "selftest_servers.py" + hook = self._hooks.get(sb.root) + if hook is not None: + sb.env["KCAP_FAKE_HOOK"] = str(hook) + if not self.live_catalogue: + sb.env["KCAP_FAKE_TUI_FROZEN"] = "1" + return [sys.executable, str(servers), "tui"] diff --git a/docs/probes/2026-09-16-skills-discovery/harness/gemini.py b/docs/probes/2026-09-16-skills-discovery/harness/gemini.py new file mode 100644 index 0000000000..e48b4c4fa4 --- /dev/null +++ b/docs/probes/2026-09-16-skills-discovery/harness/gemini.py @@ -0,0 +1,93 @@ +from __future__ import annotations + +import json +import subprocess +from pathlib import Path + +from harness.base import Adapter, AskResult, HookInfo, Session +from lib.acp_driver import AcpSession, acp_ask +from lib.isolation import Sandbox +from lib.print_driver import print_ask + + +class GeminiAdapter(Adapter): + entry = "gemini" + harness = "gemini" + binary = "gemini" + lever = "GEMINI_CLI_HOME" + credential_files = (".gemini/oauth_creds.json", ".gemini/google_accounts.json", ".gemini/installation_id") + passthrough_env = ("GEMINI_API_KEY",) + native_root = ".gemini/skills" + documented_roots = frozenset({".gemini/skills", ".agents/skills"}) + modes = ("print", "daemon", "tui") + can_resume = True + tui_exit = ("/quit\r", "\x03", "\x03") + + def real_root(self) -> Path | None: + return Path.home() + + def _settings(self, sb: Sandbox) -> Path: + return sb.config_root / ".gemini" / "settings.json" + + def prepare(self, sb: Sandbox) -> None: + self._settings(sb).parent.mkdir(parents=True, exist_ok=True) + if not self._settings(sb).exists(): + # A credential is only used once the settings name it as the auth method; an API key + # in the environment wins over the copied OAuth files. + auth = "gemini-api-key" if sb.env.get("GEMINI_API_KEY") else "oauth-personal" + self._settings(sb).write_text(json.dumps({"security": { + "auth": {"selectedType": auth}, + "folderTrust": {"enabled": False}, + }}, indent=2) + "\n") + (sb.config_root / ".gemini" / "trustedFolders.json").write_text(json.dumps({str(sb.repo): "TRUST_FOLDER"}) + "\n") + + def check_auth(self, sb: Sandbox) -> bool | None: + return (sb.config_root / ".gemini" / "oauth_creds.json").exists() or None + + def install_startup_hook(self, sb: Sandbox, script: Path) -> HookInfo: + data = json.loads(self._settings(sb).read_text() or "{}") + data.setdefault("hooks", {})["SessionStart"] = [ + {"hooks": [{"name": "probe", "type": "command", "command": str(script), "timeout": 30000}]} + ] + self._settings(sb).write_text(json.dumps(data, indent=2) + "\n") + return HookInfo(mechanism="settings.json hooks.SessionStart", config_path=str(self._settings(sb))) + + def acp_argv(self) -> list[str]: + return [self.binary_path() or self.binary, "--experimental-acp", "--skip-trust", "--approval-mode", "yolo"] + + def tui_argv(self, sb: Sandbox) -> list[str] | None: + return [self.binary_path() or self.binary, "--approval-mode", "yolo"] + + def open_session(self, sb: Sandbox, mode: str) -> Session | None: + if mode != "daemon": + return super().open_session(sb, mode) + session = AcpSession(self.acp_argv(), sb.cwd, sb.env, sb.root / "gemini-acp.stderr.log", self.turn_timeout) + session.start() + return session + + def session_id(self, res: AskResult) -> str | None: + try: + sid = json.loads(res.raw).get("sessionId") or json.loads(res.raw).get("session_id") + except (json.JSONDecodeError, AttributeError): + sid = None + # The resume flag also takes "latest", which is the session this sandbox just created. + return sid or "latest" + + def resume(self, sb: Sandbox, session_id: str, prompt: str) -> AskResult | None: + return self._print(sb, prompt, ["--resume", session_id]) + + def ask(self, sb: Sandbox, mode: str, prompt: str) -> AskResult: + if mode == "daemon": + return acp_ask(self.acp_argv(), sb.cwd, sb.env, prompt, sb.root / "gemini-acp.stderr.log", + self.turn_timeout) + return self._print(sb, prompt) + + def _print(self, sb: Sandbox, prompt: str, extra: list[str] = ()) -> AskResult: + argv = [self.binary_path() or self.binary, "-p", prompt, "-o", "json", "--approval-mode", "yolo", *extra] + return print_ask(argv, sb.cwd, sb.env, sb.root / "gemini.stderr.log", self.turn_timeout, + extract=lambda raw: json.loads(raw).get("response") or "") + + def list_catalogue(self, sb: Sandbox) -> str | None: + out = subprocess.run([self.binary_path() or self.binary, "skills", "list"], cwd=str(sb.repo), env=sb.env, + capture_output=True, text=True, timeout=120) + return out.stdout diff --git a/docs/probes/2026-09-16-skills-discovery/harness/kiro.py b/docs/probes/2026-09-16-skills-discovery/harness/kiro.py new file mode 100644 index 0000000000..1244521b91 --- /dev/null +++ b/docs/probes/2026-09-16-skills-discovery/harness/kiro.py @@ -0,0 +1,181 @@ +from __future__ import annotations + +import json +import re +import subprocess +from pathlib import Path + +from harness.base import Adapter, AskResult, ClassifiedSession, HookInfo, Session +from lib.acp_driver import AcpSession, acp_ask +from lib.isolation import Sandbox +from lib.print_driver import print_ask + + +class KiroAdapter(Adapter): + entry = "kiro" + harness = "kiro" + binary = "kiro-cli" + lever = "KIRO_HOME" + passthrough_env = ("KIRO_API_KEY",) + native_root = ".kiro/skills" + documented_roots = frozenset({".kiro/skills"}) + modes = ("print", "daemon", "tui") + can_resume = True + tui_dialogs = ((r"Yes, I accept", "\x1b[B\r"),) + tui_exit = ("/quit\r", "\x03", "\x03") + agent_name: str | None = None + # None keeps whatever the cloned default agent declares; a tuple replaces it. + agent_resources: tuple[str, ...] | None = None + + def real_root(self) -> Path | None: + return Path.home() / ".kiro" + + def check_auth(self, sb: Sandbox) -> bool | None: + # An unauthenticated `chat` opens a browser login instead of answering; whoami says first. + out = subprocess.run([self.binary_path() or self.binary, "whoami"], env=sb.env, capture_output=True, + text=True, timeout=60) + text = (out.stdout + out.stderr).lower() + return out.returncode == 0 and "not logged in" not in text and "log in" not in text + + def major(self) -> int: + m = re.search(r"(\d+)\.\d+", self.version()) + return int(m.group(1)) if m else 0 + + def _settings(self, sb: Sandbox) -> Path: + d = sb.config_root / "settings" + d.mkdir(parents=True, exist_ok=True) + return d / "cli.json" + + def _agent_file(self, sb: Sandbox, name: str, hooks: dict | None) -> Path: + """A custom agent cloned from the default one, as kcap's installer does: a minimal agent + definition has no tools, and a model that cannot read the listed SKILL.md never loads it.""" + agents = sb.config_root / "agents" + agents.mkdir(parents=True, exist_ok=True) + path = agents / f"{name}.json" + data: dict = {} + if self.binary_path(): + env = dict(sb.env, EDITOR="true", VISUAL="true") + subprocess.run([self.binary_path(), "agent", "create", name, "--from", "kiro_default"], env=env, + capture_output=True, text=True, timeout=120) + if path.exists(): + try: + data = json.loads(path.read_text()) + except json.JSONDecodeError: + data = {} + if not data: + data = {"name": name, "description": "probe agent", "tools": ["*"]} + data["name"] = name + if self.agent_resources is not None: + data["resources"] = list(self.agent_resources) + if hooks: + data["hooks"] = hooks + path.write_text(json.dumps(data, indent=2) + "\n") + return path + + def prepare(self, sb: Sandbox) -> None: + if self.agent_name: + self._agent_file(sb, self.agent_name, None) + + def install_startup_hook(self, sb: Sandbox, script: Path) -> HookInfo: + if self.major() >= 3: + hooks = sb.config_root / "hooks" + hooks.mkdir(parents=True, exist_ok=True) + path = hooks / "probe.json" + path.write_text(json.dumps({"version": "v1", "hooks": [{ + "name": "probe", "trigger": "SessionStart", + "action": {"type": "command", "command": str(script)}, "timeout": 30, "enabled": True, + }]}, indent=2) + "\n") + return HookInfo(mechanism="hooks/*.json SessionStart (cli 3.x)", config_path=str(path)) + # Hooks fire only for the active agent, so the hook rides a custom agent made the default. + name = self.agent_name or "probe" + path = self._agent_file(sb, name, {"agentSpawn": [{"command": str(script)}]}) + self._settings(sb).write_text(json.dumps({"chat.defaultAgent": name}) + "\n") + return HookInfo(mechanism="agent hooks.agentSpawn (cli 2.x)", config_path=str(path)) + + def _agent_argv(self) -> list[str]: + return ["--agent", self.agent_name] if self.agent_name else [] + + def acp_argv(self) -> list[str]: + return [self.binary_path() or self.binary, "acp", "--trust-all-tools", *self._agent_argv()] + + def tui_argv(self, sb: Sandbox) -> list[str] | None: + return [self.binary_path() or self.binary, "chat", "--trust-all-tools", *self._agent_argv()] + + def open_session(self, sb: Sandbox, mode: str) -> Session | None: + if mode != "daemon": + return super().open_session(sb, mode) + inner = AcpSession(self.acp_argv(), sb.cwd, sb.env, sb.root / "kiro-acp.stderr.log", self.turn_timeout) + inner.start() + return ClassifiedSession(inner, lambda r: classify_kiro_tools(r.raw)) + + def session_id(self, res: AskResult) -> str | None: + # Kiro's non-interactive output carries no id; its resume flag takes the most recent + # conversation of the launch directory, which is the one this sandbox just had. + return "most-recent-in-cwd" + + def resume(self, sb: Sandbox, session_id: str, prompt: str) -> AskResult | None: + return self._chat(sb, prompt, ["--resume"]) + + def ask(self, sb: Sandbox, mode: str, prompt: str) -> AskResult: + if mode == "daemon": + res = acp_ask(self.acp_argv(), sb.cwd, sb.env, prompt, + sb.root / "kiro-acp.stderr.log", self.turn_timeout) + generic = " ".join(n for n in res.notes.split() if not n.startswith("tools_used=")) + res.notes = (generic + " " + classify_kiro_tools(res.raw)).strip() + return res + return self._chat(sb, prompt) + + def _chat(self, sb: Sandbox, prompt: str, extra: list[str] = ()) -> AskResult: + argv = [self.binary_path() or self.binary, "chat", "--no-interactive", "--trust-all-tools", + *self._agent_argv(), *extra, prompt] + res = print_ask(argv, sb.cwd, sb.env, sb.root / "kiro.stderr.log", self.turn_timeout) + # Plain chat output carries the answer only, so tool use is unobservable in this mode. + res.notes = (res.notes + " tools=unobserved").strip() + return res + + +def classify_kiro_tools(raw: str) -> str: + """Kiro lists a skill with its path and the model reads that file, so a read whose location + is a SKILL.md path is the native mechanism; a shell or search tool makes a sighting + inconclusive.""" + calls: dict[str, dict] = {} + try: + frames = json.loads(raw) + except json.JSONDecodeError: + frames = [] + for f in frames: + fr = f.get("frame") or {} + if fr.get("method") != "session/update": + continue + upd = (fr.get("params") or {}).get("update") or {} + if not upd.get("sessionUpdate", "").startswith("tool_call"): + continue + call = calls.setdefault(upd.get("toolCallId") or "", {"kind": "", "paths": []}) + if upd.get("kind"): + call["kind"] = upd["kind"] + call["paths"] += [loc.get("path", "") for loc in upd.get("locations") or [] if isinstance(loc, dict)] + skill_reads = searches = other = 0 + for call in calls.values(): + if call["kind"] == "read" and any(p.endswith("SKILL.md") for p in call["paths"]): + skill_reads += 1 + elif call["kind"] in ("execute", "search", "fetch"): + searches += 1 + else: + other += 1 + return f"tools_used={searches + other} skill_reads={skill_reads} searches={searches}" + + +class KiroAgentBareAdapter(KiroAdapter): + """A cloned default agent with its resources emptied: does it still inherit the skills?""" + + entry = "kiro-agent-bare" + agent_name = "probe-bare" + agent_resources = () + + +class KiroAgentSkillsAdapter(KiroAdapter): + """A cloned default agent declaring the skills resource explicitly.""" + + entry = "kiro-agent-skills" + agent_name = "probe-skills" + agent_resources = ("skill://.kiro/skills/*/SKILL.md",) diff --git a/docs/probes/2026-09-16-skills-discovery/harness/opencode_v1.py b/docs/probes/2026-09-16-skills-discovery/harness/opencode_v1.py new file mode 100644 index 0000000000..253bf72447 --- /dev/null +++ b/docs/probes/2026-09-16-skills-discovery/harness/opencode_v1.py @@ -0,0 +1,199 @@ +from __future__ import annotations + +import json +import os +import shutil +from pathlib import Path + +from harness.base import Adapter, AskResult, ClassifiedSession, HookInfo, Session +from lib.acp_driver import AcpSession, acp_ask +from lib.hook_script import stamp_path, write_hook_script +from lib.isolation import Sandbox +from lib.print_driver import print_ask + +V1_EVENT_PLUGIN = """import {{ execFileSync }} from "node:child_process" + +export const ProbePlugin = async () => ({{ + event: async ({{ event }}: any) => {{ + if (event?.type === "session.created") {{ + try {{ execFileSync({script}, {{ stdio: "ignore" }}) }} catch {{}} + }} + }}, +}}) +""" + +V1_TRANSFORM_PLUGIN = """import {{ execFileSync }} from "node:child_process" + +export const ProbeRegisterPlugin = async () => ({{ + "experimental.chat.system.transform": async () => {{ + try {{ execFileSync({script}, {{ stdio: "ignore" }}) }} catch {{}} + }}, +}}) +""" + +SEARCH_TOOLS = ("bash", "grep", "glob", "list", "read", "webfetch") + + +class OpenCodeV1Adapter(Adapter): + entry = "opencode-v1" + harness = "opencode" + binary = "opencode" + lever = "OPENCODE_CONFIG_DIR" + native_root = ".opencode/skills" + documented_roots = frozenset({".opencode/skills", ".claude/skills", ".agents/skills"}) + modes = ("print", "daemon", "tui") + can_resume = True + tui_exit = ("\x03", "\x03", "\x04") + # Flags a version accepts only after its subcommand. + extra_argv: tuple[str, ...] = () + + def real_root(self) -> Path | None: + return Path.home() / ".config" / "opencode" + + def default_model(self) -> str | None: + if os.environ.get("KCAP_PROBE_OPENCODE_MODEL"): + return os.environ["KCAP_PROBE_OPENCODE_MODEL"] + cfg = (self.real_root() or Path()) / "opencode.json" + if cfg.exists(): + try: + return json.loads(cfg.read_text()).get("model") + except json.JSONDecodeError: + return None + return None + + def prepare(self, sb: Sandbox) -> None: + data = sb.root / "data" + (data / "opencode").mkdir(parents=True, exist_ok=True) + sb.env["XDG_DATA_HOME"] = str(data) + auth = Path.home() / ".local" / "share" / "opencode" / "auth.json" + if auth.exists(): + shutil.copy2(auth, data / "opencode" / "auth.json") + cfg = {"$schema": "https://opencode.ai/config.json"} + if self.default_model(): + cfg["model"] = self.default_model() + (sb.config_root / "opencode.json").write_text(json.dumps(cfg, indent=2) + "\n") + + def check_auth(self, sb: Sandbox) -> bool | None: + return (Path(sb.env["XDG_DATA_HOME"]) / "opencode" / "auth.json").exists() + + def _plugins(self, sb: Sandbox) -> Path: + d = sb.config_root / "plugins" + d.mkdir(parents=True, exist_ok=True) + return d + + def install_startup_hook(self, sb: Sandbox, script: Path) -> HookInfo: + path = self._plugins(sb) / "probe.ts" + path.write_text(V1_EVENT_PLUGIN.format(script=json.dumps(str(script)))) + return HookInfo(mechanism="plugin event session.created", config_path=str(path)) + + def install_registration(self, sb: Sandbox, skill_file: Path, body: str) -> HookInfo | None: + script = write_hook_script(sb.config_root, skill_file, body, stamp_path(sb.config_root)) + path = self._plugins(sb) / "probe-register.ts" + path.write_text(V1_TRANSFORM_PLUGIN.format(script=json.dumps(str(script)))) + return HookInfo(mechanism="plugin experimental.chat.system.transform", config_path=str(path)) + + def acp_argv(self) -> list[str]: + return [self.binary_path() or self.binary, "acp", *self.extra_argv] + + def tui_argv(self, sb: Sandbox) -> list[str] | None: + return [self.binary_path() or self.binary, *self.extra_argv] + + def open_session(self, sb: Sandbox, mode: str) -> Session | None: + if mode != "daemon": + return super().open_session(sb, mode) + inner = AcpSession(self.acp_argv(), sb.cwd, sb.env, sb.root / "opencode-acp.stderr.log", self.turn_timeout) + inner.start() + return ClassifiedSession(inner, lambda r: classify_acp_tools(r.raw)) + + def session_id(self, res: AskResult) -> str | None: + for line in res.raw.splitlines(): + try: + obj = json.loads(line) + except json.JSONDecodeError: + continue + for holder in (obj, obj.get("part") if isinstance(obj.get("part"), dict) else {}): + if isinstance(holder.get("sessionID"), str): + return holder["sessionID"] + return None + + def resume(self, sb: Sandbox, session_id: str, prompt: str) -> AskResult | None: + return self._print(sb, prompt, ["--session", session_id]) + + def ask(self, sb: Sandbox, mode: str, prompt: str) -> AskResult: + if mode == "daemon": + res = acp_ask(self.acp_argv(), sb.cwd, sb.env, prompt, + sb.root / "opencode-acp.stderr.log", self.turn_timeout) + generic = " ".join(n for n in res.notes.split() if not n.startswith("tools_used=")) + res.notes = (generic + " " + classify_acp_tools(res.raw)).strip() + return res + return self._print(sb, prompt) + + def _print(self, sb: Sandbox, prompt: str, extra: list[str] = ()) -> AskResult: + binary = self.binary_path() or self.binary + + def extract(raw: str) -> str: + texts = [] + for line in raw.splitlines(): + try: + obj = json.loads(line) + except json.JSONDecodeError: + continue + part = obj.get("part") if isinstance(obj.get("part"), dict) else obj + if part.get("type") == "text" and isinstance(part.get("text"), str): + texts.append(part["text"]) + return "\n".join(texts) if texts else raw + + argv = [binary, "run", *self.extra_argv, *extra, "--format", "json", prompt] + res = print_ask(argv, sb.cwd, sb.env, sb.root / "opencode.stderr.log", self.turn_timeout, extract=extract) + if "USAGE" in res.reply_text and "FLAGS" in res.reply_text: + # The CLI printed its usage instead of running: an invocation error, not an answer. + res.reply_text = "" + res.exit_code = res.exit_code or 2 + res.notes = (res.notes + " usage printed").strip() + res.notes = (res.notes + " " + classify_print_tools(res.raw)).strip() + return res + + +def classify_acp_tools(raw: str) -> str: + """OpenCode loads a listed skill through its own `skill` tool over ACP.""" + skill_loads = searches = other = 0 + try: + frames = json.loads(raw) + except json.JSONDecodeError: + frames = [] + for f in frames: + fr = f.get("frame") or {} + if fr.get("method") != "session/update": + continue + upd = (fr.get("params") or {}).get("update") or {} + if upd.get("sessionUpdate") != "tool_call": + continue + title = (upd.get("title") or "").lower() + if title == "skill" or title.startswith("loaded skill"): + skill_loads += 1 + elif any(title.startswith(t) for t in SEARCH_TOOLS) or upd.get("kind") in ("execute", "search"): + searches += 1 + else: + other += 1 + return f"tools_used={searches + other} skill_loads={skill_loads} searches={searches}" + + +def classify_print_tools(raw: str) -> str: + """The `run --format json` events name each tool part by its tool name.""" + skill_loads = searches = other = 0 + for line in raw.splitlines(): + try: + obj = json.loads(line) + except json.JSONDecodeError: + continue + part = obj.get("part") if isinstance(obj.get("part"), dict) else obj + if part.get("type") != "tool": + continue + tool = (part.get("tool") or "").lower() + if tool == "skill": + skill_loads += 1 + elif tool in SEARCH_TOOLS: + searches += 1 + else: + other += 1 + return f"tools_used={searches + other} skill_loads={skill_loads} searches={searches}" diff --git a/docs/probes/2026-09-16-skills-discovery/harness/opencode_v2.py b/docs/probes/2026-09-16-skills-discovery/harness/opencode_v2.py new file mode 100644 index 0000000000..da0b08fa22 --- /dev/null +++ b/docs/probes/2026-09-16-skills-discovery/harness/opencode_v2.py @@ -0,0 +1,127 @@ +from __future__ import annotations + +import json +import os +import subprocess +from pathlib import Path + +from harness.base import AskResult, HookInfo, Session +from harness.opencode_v1 import OpenCodeV1Adapter, classify_acp_tools +from lib.acp_driver import acp_ask +from lib.hook_script import stamp_path, write_hook_script +from lib.isolation import Sandbox + +V2_SETUP_PLUGIN = """import {{ execFileSync }} from "node:child_process" +import {{ Plugin }} from "@opencode/plugin" + +export default Plugin.define({{ + id: "probe", + async setup(ctx) {{ + try {{ execFileSync({script}, {{ stdio: "ignore" }}) }} catch {{}} + {reload} + }}, +}}) +""" + +V2_PROMPT_PLUGIN = """import {{ execFileSync }} from "node:child_process" +import {{ Plugin }} from "@opencode/plugin" + +export default Plugin.define({{ + id: "probe", + async setup(ctx) {{ + await ctx.session.hook("prompt", async () => {{ + try {{ execFileSync({script}, {{ stdio: "ignore" }}) }} catch {{}} + await ctx.skill.reload() + }}) + }}, +}}) +""" + + +class _ServiceStopSession(Session): + """The ACP client leaves the background service running, and the next sandbox would reuse it.""" + + def __init__(self, inner: Session, stop) -> None: + self.inner = inner + self.stop = stop + + def ask(self, prompt: str) -> AskResult: + return self.inner.ask(prompt) + + def reload(self) -> str | None: + return self.inner.reload() + + def close(self) -> None: + try: + self.inner.close() + finally: + self.stop() + + +class OpenCodeV2Adapter(OpenCodeV1Adapter): + entry = "opencode-v2" + # `run` takes a private server; `acp` rejects the flag and always uses the background service, + # which is started under the sandbox's config and stopped after the turn. + extra_argv = ("--standalone",) + + def binary_path(self) -> str | None: + p = Path(os.environ.get("KCAP_OPENCODE_V2_PATH") or (Path.home() / ".local" / "opencode-v2" / "bin" / "opencode")) + return str(p) if p.exists() else None + + def acp_argv(self) -> list[str]: + return [self.binary_path() or self.binary, "acp"] + + def tui_argv(self, sb: Sandbox) -> list[str] | None: + return [self.binary_path() or self.binary] + + def _service_stop(self, sb: Sandbox) -> None: + subprocess.run([self.binary_path() or self.binary, "service", "stop"], env=sb.env, + capture_output=True, text=True, timeout=60) + + def open_session(self, sb: Sandbox, mode: str) -> Session | None: + # A service left running by an earlier sandbox leaves the interactive UI drawing nothing + # at all, and serves the other sandbox's catalogue to a headless session. + self._service_stop(sb) + session = super().open_session(sb, mode) + if mode != "daemon" or session is None: + return session + return _ServiceStopSession(session, lambda: self._service_stop(sb)) + + def ask(self, sb: Sandbox, mode: str, prompt: str) -> AskResult: + if mode != "daemon": + return super().ask(sb, mode, prompt) + self._service_stop(sb) + try: + res = acp_ask(self.acp_argv(), sb.cwd, sb.env, prompt, sb.root / "opencode-acp.stderr.log", + self.turn_timeout) + finally: + self._service_stop(sb) + generic = " ".join(n for n in res.notes.split() if not n.startswith("tools_used=")) + res.notes = (generic + " " + classify_acp_tools(res.raw)).strip() + return res + + def _plugin_file(self, sb: Sandbox) -> Path: + d = self._plugins(sb) / "probe" + d.mkdir(parents=True, exist_ok=True) + return d / "index.ts" + + def install_startup_hook(self, sb: Sandbox, script: Path) -> HookInfo: + path = self._plugin_file(sb) + path.write_text(V2_SETUP_PLUGIN.format(script=json.dumps(str(script)), reload="")) + return HookInfo(mechanism="plugin setup (no reload)", config_path=str(path)) + + def install_registration(self, sb: Sandbox, skill_file: Path, body: str) -> HookInfo | None: + script = write_hook_script(sb.config_root, skill_file, body, stamp_path(sb.config_root)) + path = self._plugin_file(sb) + path.write_text(V2_SETUP_PLUGIN.format(script=json.dumps(str(script)), reload="await ctx.skill.reload()")) + return HookInfo(mechanism="plugin setup + skill.reload", config_path=str(path)) + + +class OpenCodeV2PromptHookAdapter(OpenCodeV2Adapter): + entry = "opencode-v2-prompt" + + def install_registration(self, sb: Sandbox, skill_file: Path, body: str) -> HookInfo | None: + script = write_hook_script(sb.config_root, skill_file, body, stamp_path(sb.config_root)) + path = self._plugin_file(sb) + path.write_text(V2_PROMPT_PLUGIN.format(script=json.dumps(str(script)))) + return HookInfo(mechanism="plugin prompt hook + skill.reload", config_path=str(path)) diff --git a/docs/probes/2026-09-16-skills-discovery/harness/pi.py b/docs/probes/2026-09-16-skills-discovery/harness/pi.py new file mode 100644 index 0000000000..e50b5044a6 --- /dev/null +++ b/docs/probes/2026-09-16-skills-discovery/harness/pi.py @@ -0,0 +1,167 @@ +from __future__ import annotations + +import json +import subprocess +from pathlib import Path + +from harness.base import Adapter, AskResult, ClassifiedSession, HookInfo, Session +from lib.hook_script import stamp_path, write_hook_script +from lib.isolation import Sandbox +from lib.pirpc_driver import PiRpcSession, pirpc_ask +from lib.print_driver import print_ask + +HOOK_EXT = """import {{ execFileSync }} from "node:child_process"; + +export default function (pi: any) {{ + pi.on("session_start", async () => {{ + try {{ execFileSync({script}, {{ stdio: "ignore" }}); }} catch {{}} + }}); +}} +""" + +REGISTER_EXT = """import {{ execFileSync }} from "node:child_process"; + +export default function (pi: any) {{ + pi.on("session_start", async () => {{ + try {{ execFileSync({script}, {{ stdio: "ignore" }}); }} catch {{}} + }}); + pi.on("resources_discover", async () => {{ + return {{ skillPaths: [{root}] }}; + }}); +}} +""" + + +class PiAdapter(Adapter): + entry = "pi" + harness = "pi" + binary = "pi" + lever = "PI_CODING_AGENT_DIR" + credential_files = ("auth.json", "models.json", "settings.json") + native_root = ".pi/skills" + documented_roots = frozenset({".pi/skills", ".agents/skills"}) + modes = ("print", "daemon", "tui") + can_resume = True + tui_exit = ("/quit\r", "\x03", "\x04") + # A private config root downloads its own fd and ripgrep before the prompt appears. + tui_ready = 6.0 + + def real_root(self) -> Path | None: + return Path.home() / ".pi" / "agent" + + def check_auth(self, sb: Sandbox) -> bool | None: + auth = sb.config_root / "auth.json" + if not auth.exists(): + return False + for provider in json.loads(auth.read_text()).keys(): + out = subprocess.run([self.binary_path() or self.binary, "auth", "check", "--provider", provider, "--json"], + env=sb.env, capture_output=True, text=True, timeout=60) + if out.returncode == 0: + return True + return False + + def _ext_dir(self, sb: Sandbox) -> Path: + d = sb.config_root / "extensions" + d.mkdir(parents=True, exist_ok=True) + return d + + def install_startup_hook(self, sb: Sandbox, script: Path) -> HookInfo: + ext = self._ext_dir(sb) / "probe.ts" + ext.write_text(HOOK_EXT.format(script=json.dumps(str(script)))) + return HookInfo(mechanism="extension session_start", config_path=str(ext)) + + def install_registration(self, sb: Sandbox, skill_file: Path, body: str) -> HookInfo | None: + script = write_hook_script(sb.config_root, skill_file, body, stamp_path(sb.config_root)) + root = skill_file.parent.parent + ext = self._ext_dir(sb) / "probe-register.ts" + ext.write_text(REGISTER_EXT.format(script=json.dumps(str(script)), root=json.dumps(str(root)))) + return HookInfo(mechanism="extension session_start + resources_discover skillPaths", config_path=str(ext)) + + def rpc_argv(self) -> list[str]: + return [self.binary_path() or self.binary, "--mode", "rpc", "--approve"] + + def tui_argv(self, sb: Sandbox) -> list[str] | None: + return [self.binary_path() or self.binary, "--approve"] + + def open_session(self, sb: Sandbox, mode: str) -> Session | None: + if mode != "daemon": + return super().open_session(sb, mode) + inner = PiRpcSession(self.rpc_argv(), sb.cwd, sb.env, sb.root / "pi-rpc.stderr.log", self.turn_timeout) + inner.start() + return ClassifiedSession(inner, lambda r: classify_pi_tools(frame_events(r.raw))) + + def session_id(self, res: AskResult) -> str | None: + for msg in json_lines(res.raw): + if msg.get("type") == "session" and msg.get("id"): + return msg["id"] + return None + + def resume(self, sb: Sandbox, session_id: str, prompt: str) -> AskResult | None: + return self._print(sb, prompt, ["--session", session_id]) + + def ask(self, sb: Sandbox, mode: str, prompt: str) -> AskResult: + if mode == "daemon": + res = pirpc_ask(self.rpc_argv(), sb.cwd, sb.env, prompt, + sb.root / "pi-rpc.stderr.log", self.turn_timeout) + generic = " ".join(n for n in res.notes.split() if not n.startswith("tools_used=")) + res.notes = (generic + " " + classify_pi_tools(frame_events(res.raw))).strip() + return res + return self._print(sb, prompt) + + def _print(self, sb: Sandbox, prompt: str, extra: list[str] = ()) -> AskResult: + binary = self.binary_path() or self.binary + + def extract(raw: str) -> str: + texts = [] + for line in raw.splitlines(): + try: + msg = json.loads(line) + except json.JSONDecodeError: + continue + if msg.get("type") == "message_end" and (msg.get("message") or {}).get("role") == "assistant": + texts += [p.get("text", "") for p in msg["message"].get("content") or [] + if isinstance(p, dict) and p.get("type") == "text"] + return "\n".join(texts) + + argv = [binary, "-p", "--mode", "json", "--approve", *extra, "--", prompt] + res = print_ask(argv, sb.cwd, sb.env, sb.root / "pi.stderr.log", self.turn_timeout, extract=extract) + res.notes = (res.notes + " " + classify_pi_tools(json_lines(res.raw))).strip() + return res + + +SEARCH_TOOLS = ("bash", "grep", "find", "glob", "ls") + + +def json_lines(raw: str) -> list[dict]: + out = [] + for line in raw.splitlines(): + try: + out.append(json.loads(line)) + except json.JSONDecodeError: + continue + return out + + +def frame_events(raw: str) -> list[dict]: + try: + return [f.get("frame") or {} for f in json.loads(raw)] + except (json.JSONDecodeError, TypeError, AttributeError): + return [] + + +def classify_pi_tools(events: list[dict]) -> str: + """Pi lists a skill with its path and the model reads that file, so a `read` of a SKILL.md + path is the native mechanism; a shell or search tool is what makes a sighting inconclusive.""" + skill_reads = searches = other = 0 + for msg in events: + if msg.get("type") != "tool_execution_start": + continue + name = msg.get("toolName") or "" + args = msg.get("args") or {} + if name in SEARCH_TOOLS: + searches += 1 + elif name == "read" and str(args.get("path", "")).endswith("SKILL.md"): + skill_reads += 1 + else: + other += 1 + return f"tools_used={searches + other} skill_reads={skill_reads} searches={searches}" diff --git a/docs/probes/2026-09-16-skills-discovery/lib/__init__.py b/docs/probes/2026-09-16-skills-discovery/lib/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/docs/probes/2026-09-16-skills-discovery/lib/acp_driver.py b/docs/probes/2026-09-16-skills-discovery/lib/acp_driver.py new file mode 100644 index 0000000000..5810acb4aa --- /dev/null +++ b/docs/probes/2026-09-16-skills-discovery/lib/acp_driver.py @@ -0,0 +1,170 @@ +from __future__ import annotations + +import asyncio +import json +import sys +import time +from pathlib import Path + +from harness.base import AskResult, Session +from lib.procs import kill_group + +KIT = Path(__file__).resolve().parent.parent +sys.path.insert(0, str(KIT.parent / "2026-08-04-acp-reconnect-c0")) +from acp_c0_probe import AcpClient # noqa: E402 + +INIT_PARAMS = { + "protocolVersion": 1, + "clientCapabilities": {"fs": {"readTextFile": False, "writeTextFile": False}, "terminal": False}, +} + + +class IsolatedAcpClient(AcpClient): + """AcpClient whose child sees exactly the sandbox environment, not the developer's.""" + + def __init__(self, argv, cwd, env, stderr_path): + super().__init__(argv, cwd, frames=[], phase_ref=["turn"], label="acp", stderr_path=str(stderr_path)) + self.env = env + self.stderr_f = None + + async def start(self): + self.stderr_f = open(self.stderr_path, "ab") + # Its own session, so a vendor that re-execs itself leaves no grandchild holding the + # pipes open after the child is gone (an open pipe is a shutdown that never returns). + self.proc = await asyncio.create_subprocess_exec( + *self.argv, cwd=self.cwd, env=self.env, start_new_session=True, + stdin=asyncio.subprocess.PIPE, stdout=asyncio.subprocess.PIPE, stderr=self.stderr_f) + self.reader_task = asyncio.create_task(self._read_loop()) + self.record("mark", {"event": "spawned", "pid": self.proc.pid, "argv": self.argv}) + + async def shutdown(self, hard_after=5): + # asyncio's Process.wait() only completes once every pipe is closed, so a grandchild that + # inherited stdout keeps the base class's wait from ever returning: the whole session is + # killed as soon as a graceful exit has not happened. + if self.proc is not None: + if self.proc.returncode is None: + try: + self.proc.terminate() + except ProcessLookupError: + pass + try: + await asyncio.wait_for(self.proc.wait(), hard_after) + except asyncio.TimeoutError: + kill_group(self.proc.pid) + try: + await asyncio.wait_for(self.proc.wait(), hard_after) + except asyncio.TimeoutError: + pass + kill_group(self.proc.pid) + if self.reader_task is not None: + await asyncio.wait([self.reader_task], timeout=5) + if self.stderr_f is not None: + self.stderr_f.close() + self.stderr_f = None + + +def agent_text(frames: list[dict]) -> str: + out = [] + for f in frames: + fr = f.get("frame") or {} + if fr.get("method") != "session/update": + continue + upd = ((fr.get("params") or {}).get("update") or {}) + if upd.get("sessionUpdate") not in ("agent_message_chunk", "agent_message"): + continue + content = upd.get("content") + if isinstance(content, dict) and isinstance(content.get("text"), str): + out.append(content["text"]) + elif isinstance(content, list): + out.extend(c["text"] for c in content if isinstance(c, dict) and isinstance(c.get("text"), str)) + return "".join(out) + + +def tool_calls(frames: list[dict]) -> int: + n = 0 + for f in frames: + fr = f.get("frame") or {} + if fr.get("method") != "session/update": + continue + if ((fr.get("params") or {}).get("update") or {}).get("sessionUpdate") == "tool_call": + n += 1 + return n + + +class AcpSession(Session): + """One ACP agent process holding one session across prompts.""" + + def __init__(self, argv: list[str], cwd: Path, env: dict, stderr_path: Path, timeout: float = 180.0) -> None: + self.argv = list(argv) + self.cwd = Path(cwd) + self.stderr_path = stderr_path + self.timeout = timeout + self.loop = asyncio.new_event_loop() + self.client = IsolatedAcpClient(self.argv, str(self.cwd), env, stderr_path) + self.sid: str | None = None + self.notes: list[str] = [] + self._raw_from = 0 + self.started_at = time.time() + + def start(self) -> None: + try: + self.loop.run_until_complete(self._start()) + except BaseException: + # The child is already spawned, and a caller that never got the session cannot close it. + self.close() + raise + + async def _start(self) -> None: + await self.client.start() + await self.client.request("initialize", INIT_PARAMS, timeout=90) + new = await self.client.request("session/new", {"cwd": str(self.cwd), "mcpServers": []}, timeout=120) + self.sid = (new.get("result") or {}).get("sessionId") + if not self.sid: + self.notes.append(f"session/new failed: {json.dumps(new)[:500]}") + + def ask(self, prompt: str) -> AskResult: + first = time.time() + before = len(self.client.frames) + notes = list(self.notes) + text = "" + try: + if self.sid: + resp = self.loop.run_until_complete(self.client.request( + "session/prompt", {"sessionId": self.sid, "prompt": [{"type": "text", "text": prompt}]}, + timeout=self.timeout)) + notes.append(f"stopReason={(resp.get('result') or {}).get('stopReason')}") + if "error" in resp: + notes.append(f"error={json.dumps(resp['error'])[:500]}") + text = agent_text(self.client.frames[before:]) + except Exception as ex: # noqa: BLE001 + notes.append(f"exception={ex!r}") + # A reply the agent read off disk with a tool is not a loaded skill: the count says which it was. + notes.append(f"tools_used={tool_calls(self.client.frames[before:])}") + exit_code = self.client.proc.returncode if self.client.proc is not None else None + # Each turn's raw stream starts where the previous one ended (the first includes the + # startup exchange), so a reply parsed from raw cannot credit an earlier turn's token. + raw = json.dumps(self.client.frames[self._raw_from:]) + self._raw_from = len(self.client.frames) + return AskResult(reply_text=text, raw=raw, argv=list(self.argv), + started_at=self.started_at, first_request_at=first, stderr_path=str(self.stderr_path), + exit_code=exit_code, notes=" ".join(notes)) + + def close(self) -> None: + try: + self.loop.run_until_complete(self.client.shutdown()) + except Exception as ex: # noqa: BLE001 + self.notes.append(f"close={ex!r}") + finally: + self.loop.close() + + +def acp_ask(argv: list[str], cwd: Path, env: dict, prompt: str, stderr_path: Path, timeout: float = 180.0) -> AskResult: + session = AcpSession(argv, cwd, env, stderr_path, timeout) + try: + try: + session.start() + except Exception as ex: # noqa: BLE001 + session.notes.append(f"exception={ex!r}") + return session.ask(prompt) + finally: + session.close() diff --git a/docs/probes/2026-09-16-skills-discovery/lib/appserver_driver.py b/docs/probes/2026-09-16-skills-discovery/lib/appserver_driver.py new file mode 100644 index 0000000000..fcb391db2f --- /dev/null +++ b/docs/probes/2026-09-16-skills-discovery/lib/appserver_driver.py @@ -0,0 +1,165 @@ +from __future__ import annotations + +import json +import time +from pathlib import Path + +from harness.base import AskResult, Session +from lib.jsonl_child import JsonlChild + + +class _Rpc: + def __init__(self, child: JsonlChild) -> None: + self.child = child + self.next_id = 0 + self.notifications: list[dict] = [] + + def request(self, method: str, params: dict, timeout: float) -> dict: + self.next_id += 1 + rid = self.next_id + self.child.send({"jsonrpc": "2.0", "id": rid, "method": method, "params": params}) + deadline = time.time() + timeout + while time.time() < deadline: + msg = self.child.recv(max(0.1, deadline - time.time())) + if msg is None: + raise ConnectionError(f"EOF waiting for {method}") + if msg.get("id") == rid and "method" not in msg: + return msg + self._absorb(msg) + raise TimeoutError(f"no response to {method} within {timeout}s") + + def _absorb(self, msg: dict) -> None: + if "method" in msg and "id" in msg: + self.child.send({"jsonrpc": "2.0", "id": msg["id"], "error": {"code": -32601, "message": "unsupported"}}) + elif "method" in msg: + self.notifications.append(msg) + + def wait_notification(self, method: str, timeout: float) -> dict | None: + deadline = time.time() + timeout + while time.time() < deadline: + msg = self.child.recv(max(0.1, deadline - time.time())) + if msg is None: + return None + self._absorb(msg) + if msg.get("method") == method: + return msg + return None + + +def _toml_string(s: str) -> str: + return json.dumps(s) + + +def hook_state_override(hooks: list[dict]) -> str | None: + untrusted = [h for h in hooks if h.get("trustStatus") != "trusted" and h.get("currentHash") and h.get("key")] + if not untrusted: + return None + entries = ",".join(f"{_toml_string(h['key'])}={{trusted_hash={_toml_string(h['currentHash'])}}}" for h in untrusted) + return f"hooks.state={{{entries}}}" + + +class AppServerSession(Session): + def __init__(self, binary: str, cwd: Path, env: dict, stderr_path: Path, timeout: float = 180.0, + extra_argv: list[str] = ()) -> None: + self.argv = [binary, "app-server", *extra_argv] + self.cwd = Path(cwd) + self.env = env + self.stderr_path = stderr_path + self.timeout = timeout + self.child = JsonlChild(self.argv, self.cwd, env, stderr_path) + self.rpc: _Rpc | None = None + self.tid: str | None = None + self.notes: list[str] = [] + self._raw_from = 0 + self.prior_frames: list[dict] = [] + self.started_at = time.time() + + def start(self) -> None: + try: + self._start() + except BaseException: + # The child is already spawned, and a caller that never got the session cannot close it. + self.close() + raise + + def _start(self) -> None: + self.child.start() + self.rpc = _Rpc(self.child) + init = {"clientInfo": {"name": "kcap-probe", "version": "1"}, "capabilities": {}} + self.rpc.request("initialize", init, 60) + listed = self.rpc.request("hooks/list", {}, 60).get("result") or {} + # The app-server groups hooks per cwd under `data`; a flat `hooks` list is kept for safety. + hooks = list(listed.get("hooks") or []) + for group in listed.get("data") or []: + hooks += list(group.get("hooks") or []) + override = hook_state_override(hooks) + if override: + self.child.stop() + self.prior_frames = list(self.child.frames) + self.argv = [*self.argv, "-c", override] + self.child = JsonlChild(self.argv, self.cwd, self.env, self.stderr_path) + self.child.start() + self.rpc = _Rpc(self.child) + self.rpc.request("initialize", init, 60) + self.notes.append("hook_trust=seeded") + elif not hooks: + self.notes.append("hook_trust=none") + elif any(h.get("trustStatus") == "trusted" for h in hooks): + self.notes.append("hook_trust=trusted") + else: + self.notes.append("hook_trust=untrusted-unseedable") + thread = self.rpc.request("thread/start", { + "cwd": str(self.cwd), "sandbox": "read-only", "approvalPolicy": "never", "approvalsReviewer": "user", + }, 120) + self.tid = ((thread.get("result") or {}).get("thread") or {}).get("id") + if not self.tid: + self.notes.append(f"thread/start failed: {json.dumps(thread)[:500]}") + + def ask(self, prompt: str) -> AskResult: + first = time.time() + notes = list(self.notes) + text, tools = "", 0 + before = len(self.rpc.notifications) if self.rpc is not None else 0 + try: + if self.tid and self.rpc is not None: + self.rpc.request("turn/start", { + "threadId": self.tid, "input": [{"type": "text", "text": prompt}], + "sandboxPolicy": {"type": "readOnly"}, "approvalPolicy": "never", "approvalsReviewer": "user", + }, self.timeout) + done = self.rpc.wait_notification("turn/completed", self.timeout) + notes.append(f"turn={(((done or {}).get('params') or {}).get('turn') or {}).get('status')}") + fresh = self.rpc.notifications[before:] + items = [(n.get("params") or {}).get("item") or {} for n in fresh if n.get("method") == "item/completed"] + completed = [i["text"] for i in items if i.get("type") == "agentMessage" and isinstance(i.get("text"), str)] + tools = sum(1 for i in items + if i.get("type") and i["type"] not in ("agentMessage", "reasoning", "userMessage")) + deltas = [n["params"].get("delta", "") for n in fresh if n.get("method") == "item/agentMessage/delta"] + text = "\n".join(completed) if completed else "".join(deltas) + except Exception as ex: # noqa: BLE001 + notes.append(f"exception={ex!r}") + # A reply the agent read off disk with a tool is not a loaded skill: the count says which it was. + notes.append(f"tools_used={tools}") + # Each turn's raw stream starts where the previous one ended (the first includes the + # startup exchange), so a reply parsed from raw cannot credit an earlier turn's token. + stream = self.prior_frames + self.child.frames + raw = json.dumps(stream[self._raw_from:]) + self._raw_from = len(stream) + return AskResult(reply_text=text, raw=raw, argv=list(self.argv), + started_at=self.started_at, first_request_at=first, stderr_path=str(self.stderr_path), + exit_code=self.child.returncode, notes=" ".join(notes)) + + def close(self) -> None: + self.child.stop() + + +def appserver_ask(binary: str, cwd: Path, env: dict, prompt: str, stderr_path: Path, + timeout: float = 180.0, extra_argv: list[str] = ()) -> AskResult: + session = AppServerSession(binary, cwd, env, stderr_path, timeout, extra_argv) + try: + try: + session.start() + except Exception as ex: # noqa: BLE001 + session.notes.append(f"exception={ex!r}") + return session.ask(prompt) + finally: + session.close() diff --git a/docs/probes/2026-09-16-skills-discovery/lib/git_exclusion.py b/docs/probes/2026-09-16-skills-discovery/lib/git_exclusion.py new file mode 100644 index 0000000000..d63d2f6de3 --- /dev/null +++ b/docs/probes/2026-09-16-skills-discovery/lib/git_exclusion.py @@ -0,0 +1,46 @@ +from __future__ import annotations + +from pathlib import Path + +from lib.isolation import git + +EXCLUSIONS = ("none", "gitignore", "info-exclude") + + +def apply(repo: Path, exclusion: str, rel_dir: str) -> Path | None: + pattern = f"/{rel_dir.strip('/')}/\n" + if exclusion == "none": + return None + if exclusion == "gitignore": + target = repo / ".gitignore" + with target.open("a") as f: + f.write(pattern) + git(repo, "add", ".gitignore") + git(repo, "commit", "-q", "-m", "ignore probe skill") + return target + if exclusion == "info-exclude": + raw = git(repo, "rev-parse", "--git-path", "info/exclude").strip() + target = Path(raw) if Path(raw).is_absolute() else repo / raw + target = target.resolve() + target.parent.mkdir(parents=True, exist_ok=True) + with target.open("a") as f: + f.write(pattern) + return target + raise ValueError(exclusion) + + +def assert_untracked_state(repo: Path, rel_dir: str, exclusion: str) -> str: + # An empty status is the pass condition for the excluded arms, and a skill that was never + # written produces exactly that: prove the file is there before reading anything into it. + target = repo / rel_dir + if not target.exists(): + raise AssertionError(f"{rel_dir} does not exist under {repo}") + if target.is_dir() and not any(p.is_file() for p in target.rglob("*")): + raise AssertionError(f"{rel_dir} holds no file") + out = git(repo, "status", "--porcelain", "--untracked-files=all", "--", rel_dir) + if exclusion == "none": + if "??" not in out: + raise AssertionError(f"expected an untracked entry for {rel_dir}, got: {out!r}") + elif out.strip(): + raise AssertionError(f"expected {rel_dir} to be excluded under {exclusion}, got: {out!r}") + return out diff --git a/docs/probes/2026-09-16-skills-discovery/lib/hook_script.py b/docs/probes/2026-09-16-skills-discovery/lib/hook_script.py new file mode 100644 index 0000000000..9d30a910a7 --- /dev/null +++ b/docs/probes/2026-09-16-skills-discovery/lib/hook_script.py @@ -0,0 +1,55 @@ +from __future__ import annotations + +import json +from pathlib import Path + + +def stamp_path(config_root: Path) -> Path: + return config_root / "probe-hook-fired.json" + + +def write_hook_script(config_root: Path, skill_file: Path, body: str, stamp: Path, delete: bool = False, + delete_path: Path | None = None) -> Path: + if body and not body.endswith("\n"): + body += "\n" # the heredoc terminator must start its own line + if delete: + action = f"rm -rf '{delete_path or skill_file.parent}'\n" + else: + action = ( + f"mkdir -p '{skill_file.parent}'\n" + f"cat > '{skill_file}' <<'KCAP_PROBE_EOF'\n{body}KCAP_PROBE_EOF\n" + ) + script = config_root / "probe-hook.sh" + script.write_text( + "#!/bin/sh\n" + "set -eu\n" + f"{action}" + f"printf '{{\"fired_at\": %s, \"pid\": %s}}\\n' \"$(date +%s)\" \"$$\" > '{stamp}'\n" + # A vendor that never closes the hook's stdin must not wedge the launch: capture briefly. + # Backgrounded commands get /dev/null on fd 0 unless explicitly redirected, so the real + # stdin is saved to fd 3 first and handed to the background reader from there. A vendor + # that hands the hook no stdin at all must not abort the script under set -e. A terminal + # is never read: its input is the interactive UI's, and a hook that consumes it leaves the + # vendor waiting for a keystroke it will never see. + # `exec` is a special builtin: a redirection it cannot perform ends the shell outright, + # so a subshell proves fd 0 can be duplicated before the real one does it. + "if [ ! -t 0 ] && ( exec 3<&0 ) 2>/dev/null; then\n" + " exec 3<&0\n" + f" ( cat <&3 > '{stamp}.stdin' ) & cat_pid=$!\n" + " sleep 2\n" + " kill $cat_pid 2>/dev/null || true\n" + "fi\n" + "exit 0\n" + ) + script.chmod(0o755) + return script + + +def read_stamp(stamp: Path) -> dict | None: + if not stamp.exists(): + return None + data = json.loads(stamp.read_text()) + # `date +%s` resolves to the second, which is too coarse to order the hook against the first + # request; the stamp file's own mtime is not. + data["fired_at_mtime"] = stamp.stat().st_mtime + return data diff --git a/docs/probes/2026-09-16-skills-discovery/lib/isolation.py b/docs/probes/2026-09-16-skills-discovery/lib/isolation.py new file mode 100644 index 0000000000..34d2e4b3ba --- /dev/null +++ b/docs/probes/2026-09-16-skills-discovery/lib/isolation.py @@ -0,0 +1,99 @@ +from __future__ import annotations + +import os +import shutil +import subprocess +import tempfile +from dataclasses import dataclass, field +from pathlib import Path + +ENV_ALLOWLIST = ("PATH", "TERM", "LANG", "LC_ALL", "LC_CTYPE", "TMPDIR", "SHELL", "USER", "LOGNAME") +GIT_ENV_ALLOWLIST = ("PATH", "HOME", "TMPDIR", "LANG", "LC_ALL", "LC_CTYPE", "USER", "LOGNAME") + + +def git_env() -> dict[str, str]: + """The sandbox's own Git environment: a developer's global core.excludesFile would otherwise + hide the probe skill from `git status` and turn a real exclusion result into a false one.""" + env = {k: os.environ[k] for k in GIT_ENV_ALLOWLIST if k in os.environ} + env["GIT_CONFIG_GLOBAL"] = "/dev/null" + env["GIT_CONFIG_SYSTEM"] = "/dev/null" + env["GIT_TERMINAL_PROMPT"] = "0" + return env + + +def git(repo: Path, *args: str, env: dict | None = None) -> str: + return subprocess.run( + ["git", "-C", str(repo), *args], check=True, capture_output=True, text=True, + env=env if env is not None else git_env(), + ).stdout + + +@dataclass +class Sandbox: + root: Path + repo: Path + config_root: Path + env: dict[str, str] + keep: bool = False + # Logs already copied out of this sandbox, source path to destination, so a sandbox that + # yields several rows copies each log once. + copied_logs: dict[str, str] = field(default_factory=dict) + # Where the vendor is launched: the repo unless an arm moves it into a subdirectory or a + # linked worktree. Hooks and plugins installed per project stay under `repo`. + cwd: Path = None # type: ignore[assignment] + + def __post_init__(self) -> None: + if self.cwd is None: + self.cwd = self.repo + + def cleanup(self) -> None: + if not self.keep: + shutil.rmtree(self.root, ignore_errors=True) + + +def new_sandbox( + lever: str, + real_root: Path | None, + credential_files: list[str], + passthrough_env: list[str] = (), + extra_env: dict[str, str] | None = None, + keep: bool = False, + base: Path | None = None, +) -> Sandbox: + root = Path(tempfile.mkdtemp(prefix="skprobe-", dir=base)).resolve() + repo = root / "repo" + repo.mkdir() + git(repo, "init", "-q", "-b", "main") + git(repo, "config", "user.email", "probe@example.invalid") + git(repo, "config", "user.name", "probe") + (repo / "README.md").write_text("probe repo\n") + git(repo, "add", "README.md") + git(repo, "commit", "-q", "-m", "init") + + config_root = root / "config" + config_root.mkdir() + if real_root is not None: + for rel in credential_files: + src = real_root / rel + if src.is_file(): + dst = config_root / rel + dst.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(src, dst) + + env = {k: os.environ[k] for k in ENV_ALLOWLIST if k in os.environ} + for k in passthrough_env: + if k in os.environ: + env[k] = os.environ[k] + env["HOME"] = os.environ.get("HOME", str(root)) + env[lever] = str(config_root) + if extra_env: + env.update(extra_env) + return Sandbox(root=root, repo=repo, config_root=config_root, env=env, keep=keep) + + +def add_worktree(sb: Sandbox, name: str = "wt-b") -> Path: + """A linked worktree beside the repo on its own branch: its `.git` is a file pointing into the + main checkout, and git resolves its `info/exclude` to the shared common directory.""" + path = sb.root / name + git(sb.repo, "worktree", "add", "-q", "-b", name, str(path)) + return path.resolve() diff --git a/docs/probes/2026-09-16-skills-discovery/lib/jsonl_child.py b/docs/probes/2026-09-16-skills-discovery/lib/jsonl_child.py new file mode 100644 index 0000000000..b99f115d3f --- /dev/null +++ b/docs/probes/2026-09-16-skills-discovery/lib/jsonl_child.py @@ -0,0 +1,94 @@ +from __future__ import annotations + +import json +import queue +import subprocess +import threading +import time +from pathlib import Path + +from lib.procs import kill_group + + +class JsonlChild: + def __init__(self, argv: list[str], cwd: Path, env: dict, stderr_path: Path) -> None: + self.argv = list(argv) + self.cwd = str(cwd) + self.env = env + self.stderr_path = stderr_path + self.frames: list[dict] = [] + self._q: queue.Queue = queue.Queue() + self.proc: subprocess.Popen | None = None + self._err = None + + def start(self) -> None: + self._err = open(self.stderr_path, "ab") + try: + # Its own session, so a grandchild left behind by a self-re-execing vendor can be + # killed with it instead of holding stdout open forever. + self.proc = subprocess.Popen(self.argv, cwd=self.cwd, env=self.env, stdin=subprocess.PIPE, + stdout=subprocess.PIPE, stderr=self._err, text=True, bufsize=1, + start_new_session=True) + except OSError: + self._err.close() + self._err = None + raise + self._reader_thread = threading.Thread(target=self._reader, daemon=True) + self._reader_thread.start() + + def _reader(self) -> None: + assert self.proc and self.proc.stdout + for line in self.proc.stdout: + line = line.strip() + if not line: + continue + try: + obj = json.loads(line) + except json.JSONDecodeError: + obj = {"unparseable": line[:2000]} + self.frames.append({"t": time.time(), "dir": "in", "frame": obj}) + self._q.put(obj) + self._q.put(None) + + def send(self, obj: dict) -> None: + assert self.proc and self.proc.stdin + self.frames.append({"t": time.time(), "dir": "out", "frame": obj}) + self.proc.stdin.write(json.dumps(obj) + "\n") + self.proc.stdin.flush() + + def recv(self, timeout: float) -> dict | None: + try: + return self._q.get(timeout=timeout) + except queue.Empty: + raise TimeoutError(f"no frame within {timeout}s") + + def close_stdin(self) -> None: + if self.proc and self.proc.stdin: + try: + self.proc.stdin.close() + except OSError: + pass + + def stop(self, grace: float = 5.0) -> None: + if not self.proc: + if self._err is not None: + self._err.close() + self._err = None + return + self.close_stdin() + try: + self.proc.wait(timeout=grace) + except subprocess.TimeoutExpired: + self.proc.kill() + self.proc.wait() + kill_group(self.proc.pid) + self._reader_thread.join(timeout=5) + if self.proc.stdout: + self.proc.stdout.close() + if self._err is not None: + self._err.close() + self._err = None + + @property + def returncode(self) -> int | None: + return self.proc.returncode if self.proc else None diff --git a/docs/probes/2026-09-16-skills-discovery/lib/pirpc_driver.py b/docs/probes/2026-09-16-skills-discovery/lib/pirpc_driver.py new file mode 100644 index 0000000000..20d7e80a94 --- /dev/null +++ b/docs/probes/2026-09-16-skills-discovery/lib/pirpc_driver.py @@ -0,0 +1,86 @@ +from __future__ import annotations + +import json +import time +from pathlib import Path + +from harness.base import AskResult, Session +from lib.jsonl_child import JsonlChild + + +class PiRpcSession(Session): + def __init__(self, argv: list[str], cwd: Path, env: dict, stderr_path: Path, timeout: float = 180.0) -> None: + self.argv = list(argv) + self.stderr_path = stderr_path + self.timeout = timeout + self.child = JsonlChild(self.argv, cwd, env, stderr_path) + self.next_id = 0 + self.notes: list[str] = [] + self._raw_from = 0 + self.started_at = time.time() + + def start(self) -> None: + try: + self.child.start() + except BaseException: + self.close() + raise + + def ask(self, prompt: str) -> AskResult: + first = time.time() + texts, notes = [], list(self.notes) + tools = 0 + self.next_id += 1 + try: + self.child.send({"id": str(self.next_id), "type": "prompt", "message": prompt, "streamingBehavior": "followUp"}) + deadline = time.time() + self.timeout + while time.time() < deadline: + msg = self.child.recv(max(0.1, deadline - time.time())) + if msg is None: + notes.append("eof before agent_settled") + break + t = msg.get("type") + if t == "message_end" and (msg.get("message") or {}).get("role") == "assistant": + for part in (msg["message"].get("content") or []): + if isinstance(part, dict) and part.get("type") == "text": + texts.append(part.get("text", "")) + if msg["message"].get("stopReason") == "error": + notes.append(f"error={msg['message'].get('errorMessage')}") + elif t == "tool_execution_end": + # A tool call also appears as a toolCall part of the message; count it once. + tools += 1 + elif t == "response" and msg.get("success") is False: + notes.append(f"prompt rejected: {json.dumps(msg)[:300]}") + elif t == "extension_ui_request": + notes.append(f"blocked on ui request {msg.get('method')}") + break + elif t == "agent_settled": + break + else: + notes.append("timeout") + except Exception as ex: # noqa: BLE001 + notes.append(f"exception={ex!r}") + # A reply the agent read off disk with a tool is not a loaded skill: the count says which it was. + notes.append(f"tools_used={tools}") + # Each turn's raw stream starts where the previous one ended, so a reply parsed from raw + # cannot credit an earlier turn's token. + raw = json.dumps(self.child.frames[self._raw_from:]) + self._raw_from = len(self.child.frames) + return AskResult(reply_text="\n".join(texts), raw=raw, argv=list(self.argv), + started_at=self.started_at, first_request_at=first, stderr_path=str(self.stderr_path), + exit_code=self.child.returncode, notes=" ".join(notes)) + + def close(self) -> None: + self.child.stop() + + +def pirpc_ask(argv: list[str], cwd: Path, env: dict, prompt: str, stderr_path: Path, timeout: float = 180.0) -> AskResult: + session = PiRpcSession(argv, cwd, env, stderr_path, timeout) + try: + try: + session.start() + except Exception as ex: # noqa: BLE001 + session.notes.append(f"exception={ex!r}") + return session.ask(prompt) + finally: + session.close() diff --git a/docs/probes/2026-09-16-skills-discovery/lib/print_driver.py b/docs/probes/2026-09-16-skills-discovery/lib/print_driver.py new file mode 100644 index 0000000000..3a93b14953 --- /dev/null +++ b/docs/probes/2026-09-16-skills-discovery/lib/print_driver.py @@ -0,0 +1,45 @@ +from __future__ import annotations + +import subprocess +import time +from pathlib import Path +from typing import Callable + +from harness.base import AskResult +from lib.procs import kill_group + + +def print_ask( + argv: list[str], cwd: Path, env: dict, stderr_path: Path, timeout: float, + stdin_text: str | None = None, extract: Callable[[str], str] | None = None, +) -> AskResult: + started = time.time() + notes = "" + with stderr_path.open("ab") as err: + # Its own session: a vendor that re-execs leaves a grandchild holding stdout, and waiting + # on the pipe after killing only the child never returns. + proc = subprocess.Popen( + argv, cwd=str(cwd), env=env, stdout=subprocess.PIPE, stderr=err, text=True, + stdin=subprocess.PIPE if stdin_text is not None else subprocess.DEVNULL, + start_new_session=True, + ) + try: + raw, _ = proc.communicate(input=stdin_text, timeout=timeout) + code = proc.returncode + except subprocess.TimeoutExpired: + kill_group(proc.pid) + proc.kill() + try: + raw, _ = proc.communicate(timeout=10) + except subprocess.TimeoutExpired: + raw = "" + code, notes = None, "timeout" + reply = raw + if extract is not None: + try: + reply = extract(raw) + except Exception as ex: # noqa: BLE001 + # A silent fallback to the raw stream reads as a vendor that answered in plain text. + reply, notes = raw, (notes + f" extract failed: {ex!r}").strip() + return AskResult(reply_text=reply, raw=raw, argv=list(argv), started_at=started, + first_request_at=started, stderr_path=str(stderr_path), exit_code=code, notes=notes) diff --git a/docs/probes/2026-09-16-skills-discovery/lib/probe_skill.py b/docs/probes/2026-09-16-skills-discovery/lib/probe_skill.py new file mode 100644 index 0000000000..75aae186ed --- /dev/null +++ b/docs/probes/2026-09-16-skills-discovery/lib/probe_skill.py @@ -0,0 +1,140 @@ +from __future__ import annotations + +import re +import secrets +from dataclasses import dataclass +from pathlib import Path + +TOKEN_RE = re.compile(r"PROBE-BODY-([0-9a-f]{12})") +NAME_RE = re.compile(r"kcap-probe-[0-9a-f]{6}") +NO_SKILL = "NO-SKILL" + + +@dataclass(frozen=True) +class ProbeSkill: + nonce: str + token: str + + @staticmethod + def fresh() -> "ProbeSkill": + return ProbeSkill(secrets.token_hex(3), secrets.token_hex(6)) + + @property + def name(self) -> str: + return f"kcap-probe-{self.nonce}" + + @property + def body_token(self) -> str: + return f"PROBE-BODY-{self.token}" + + @property + def description(self) -> str: + return "Answers a question about the probe token for this repository." + + def render(self) -> str: + return ( + "---\n" + f"name: {self.name}\n" + f"description: {self.description}\n" + "---\n" + "When asked for the probe token, reply with exactly this token and nothing else:\n" + f"{self.body_token}\n" + ) + + def variant(self) -> "ProbeSkill": + """The same skill name with a fresh token: what an in-place rewrite of the body looks like.""" + return ProbeSkill(self.nonce, secrets.token_hex(6)) + + +def write_skill(root: Path, skill: ProbeSkill, flat: bool = False) -> Path: + if flat: + root.mkdir(parents=True, exist_ok=True) + path = root / f"{skill.name}.md" + else: + (root / skill.name).mkdir(parents=True, exist_ok=True) + path = root / skill.name / "SKILL.md" + path.write_text(skill.render()) + return path + + +# A harness may load a listed skill's body by reading its own file; what the prompt forbids is +# hunting for a file that was never listed, which would pass discovery off as a search. +NO_SEARCH = ( + "Do not search the file system or guess where it might be: only a skill already listed for " + "you counts, and reading that listed skill's own file is fine." +) + + +def single_prompt(skill: ProbeSkill, again: bool = False) -> str: + # A later turn of the same session asks for the token the skill carries now, so a session that + # kept the first answer in its context has to look again to be right. + use = "Use it again and reply with the probe token it contains now" if again else \ + "Use it and reply with only the probe token it contains" + return ( + f"You have a skill named {skill.name}. {use}. {NO_SEARCH} If no such skill is listed for " + f"you, reply with exactly {NO_SKILL}." + ) + + +def multi_prompt() -> str: + return ( + "List every skill listed for you whose name starts with kcap-probe-. For each, reply " + "with = on its own line, reading the token from the skill body. " + f"{NO_SEARCH} If there are none, reply with exactly {NO_SKILL}." + ) + + +# The echoed prompt spells the form with angle brackets, so only a real value matches; a tool +# panel that shows the skill file carries the token without the prefix and does not match either. +TUI_REPLY_RE = re.compile(r"PROBE-REPLY:\s*\**\s*(PROBE-BODY-[0-9a-f]{12}|NO-SKILL)\b") +MARK_RE = re.compile(r"PROBE-MARK-[0-9a-f]{4}") + + +def tui_prompt(skill: ProbeSkill, again: bool = False) -> str: + # A vendor that redraws its whole transcript still shows the previous turn's answer, so a + # later turn asks for a mark of its own and only a line carrying it counts. + if not again: + return ( + f"You have a skill named {skill.name}. Use it and read the probe token it contains, " + f"then reply with one line of the form PROBE-REPLY: , where is that " + f"token. {NO_SEARCH} If no such skill is listed for you, the value is {NO_SKILL}." + ) + mark = f"PROBE-MARK-{secrets.token_hex(2)}" + return ( + f"You have a skill named {skill.name}. Use it again and read the probe token it contains " + f"now, then reply with one line of the form PROBE-REPLY: {mark} , where is " + f"that token. {NO_SEARCH} If no such skill is listed for you, the value is {NO_SKILL}." + ) + + +def reply_re(mark: str | None) -> re.Pattern: + if not mark: + return TUI_REPLY_RE + return re.compile(rf"PROBE-REPLY:\s*\**\s*{re.escape(mark)}\s*\**\s*(PROBE-BODY-[0-9a-f]{{12}}|NO-SKILL)\b") + + +def extract_tui_reply(screen: str, prompt: str = "") -> str: + """The reply lines on this screen, keeping only those that carry the prompt's own mark.""" + found = MARK_RE.search(prompt) + mark = found.group(0) if found else None + return "\n".join(f"PROBE-REPLY: {m.group(1)}" for m in reply_re(mark).finditer(screen)) + + +@dataclass(frozen=True) +class Reply: + tokens: frozenset[str] + skill_named: bool + no_skill: bool + + +def parse_reply(reply_text: str, raw: str, name: str | None = None) -> Reply: + # The reply decides; the raw event log is consulted only when no reply text was extracted, + # because a tool-call frame that echoes SKILL.md would otherwise count as a loaded skill. + found = TOKEN_RE.findall(reply_text) if reply_text.strip() else TOKEN_RE.findall(raw) + # A reply that lists some other probe skill has not named this one. + named = (name in reply_text) if name else NAME_RE.search(reply_text) is not None + return Reply( + tokens=frozenset(found), + skill_named=named, + no_skill=NO_SKILL in reply_text, + ) diff --git a/docs/probes/2026-09-16-skills-discovery/lib/procs.py b/docs/probes/2026-09-16-skills-discovery/lib/procs.py new file mode 100644 index 0000000000..b9b2382991 --- /dev/null +++ b/docs/probes/2026-09-16-skills-discovery/lib/procs.py @@ -0,0 +1,13 @@ +from __future__ import annotations + +import os +import signal + + +def kill_group(pid: int) -> None: + """Kill every process in the child's own session: a vendor that re-execs itself leaves a + grandchild holding the pipes, and a pipe still open is a read that never returns.""" + try: + os.killpg(pid, signal.SIGKILL) + except (ProcessLookupError, PermissionError): + pass diff --git a/docs/probes/2026-09-16-skills-discovery/lib/pty_driver.py b/docs/probes/2026-09-16-skills-discovery/lib/pty_driver.py new file mode 100644 index 0000000000..8a5903c7b3 --- /dev/null +++ b/docs/probes/2026-09-16-skills-discovery/lib/pty_driver.py @@ -0,0 +1,411 @@ +from __future__ import annotations + +import codecs +import fcntl +import os +import pty +import re +import struct +import subprocess +import termios +import threading +import time +from pathlib import Path + +from harness.base import AskResult, Session +from lib.probe_skill import extract_tui_reply +from lib.procs import kill_group + +ANSI_RE = re.compile(r"\x1b\[[0-?]*[ -/]*[@-~]|\x1b\][^\x07\x1b]*(?:\x07|\x1b\\)|\x1b[@-Z\\-_]|\r") +CSI_RE = re.compile(r"\x1b\[([0-?]*)([ -/]*)([@-~])") +OSC_RE = re.compile(r"\x1b\][^\x07\x1b]*(?:\x07|\x1b\\)") +# Cursor movement is how some renderers space words apart; dropping it would glue them together. +CURSOR_FORWARD_RE = re.compile(r"\x1b\[(\d*)C") +CURSOR_COLUMN_RE = re.compile(r"\x1b\[\d*G") +KEY_RE = re.compile(r"\x1b\[[0-?]*[ -/]*[@-~]|\x1b.|.", re.S) + + +def strip_ansi(text: str) -> str: + text = CURSOR_FORWARD_RE.sub(lambda m: " " * int(m.group(1) or 1), text) + text = CURSOR_COLUMN_RE.sub(" ", text) + return ANSI_RE.sub("", text) + + +class Terminal: + """A small terminal emulator. A coding agent's UI redraws in place and streams its answer + beside a spinner, so the bytes read back are not a screen until the cursor movements have been + applied to a grid: concatenated raw, one status redraw lands in the middle of the reply.""" + + def __init__(self, rows: int = 50, cols: int = 200) -> None: + self.rows, self.cols = rows, cols + self.clear() + + def clear(self) -> None: + self.grid = [[" "] * self.cols for _ in range(self.rows)] + self.history: list[str] = [] + self.r = self.c = 0 + self._saved = (0, 0) + self._pending = "" + + def text(self) -> str: + return "\n".join(self.history + ["".join(row).rstrip() for row in self.grid]) + + def feed(self, data: str) -> None: + # A read ends wherever the operating system split it, so a control sequence cut in half is + # held back until its tail arrives; rendered as text it would land in the middle of a reply. + data = self._pending + data + self._pending = "" + cut = _incomplete_at_end(data) + if cut is not None: + self._pending = data[cut:] + data = data[:cut] + i, n = 0, len(data) + while i < n: + ch = data[i] + if ch == "\x1b": + csi = CSI_RE.match(data, i) + if csi: + self._csi(csi.group(1), csi.group(3)) + i = csi.end() + continue + osc = OSC_RE.match(data, i) + if osc: + i = osc.end() + continue + nxt = data[i + 1] if i + 1 < n else "" + if nxt == "7": + self._saved = (self.r, self.c) + elif nxt == "8": + self.r, self.c = self._saved + elif nxt == "M": + self._reverse_index() + elif nxt == "D": + self._index() + i += 2 + continue + i += 1 + if ch == "\r": + self.c = 0 + elif ch == "\n": + self._index() + self.c = 0 + elif ch == "\b": + self.c = max(0, self.c - 1) + elif ch == "\t": + self.c = min(self.cols - 1, (self.c // 8 + 1) * 8) + elif ch >= " " or ch == "\x7f": + self._put(ch) + + def _put(self, ch: str) -> None: + if self.c >= self.cols: + self.c = 0 + self._index() + self.grid[self.r][self.c] = ch + self.c += 1 + + def _index(self) -> None: + if self.r + 1 >= self.rows: + # The line leaving the screen is the transcript a scrolling UI would keep. + self.history.append("".join(self.grid.pop(0)).rstrip()) + self.grid.append([" "] * self.cols) + else: + self.r += 1 + + def _reverse_index(self) -> None: + if self.r == 0: + self.grid.insert(0, [" "] * self.cols) + self.grid.pop() + else: + self.r -= 1 + + def _blank(self, row: int, start: int, end: int) -> None: + for col in range(max(0, start), min(self.cols, end)): + self.grid[row][col] = " " + + def _csi(self, params: str, final: str) -> None: + if params.startswith("?"): + if final in ("h", "l") and params in ("?1049", "?47", "?1047"): + self.clear() + return + nums = [int(p) if p.isdigit() else 0 for p in params.split(";")] if params else [] + + def n(idx: int = 0, default: int = 1) -> int: + return nums[idx] if idx < len(nums) and nums[idx] else default + + if final == "A": + self.r = max(0, self.r - n()) + elif final in ("B", "e"): + self.r = min(self.rows - 1, self.r + n()) + elif final in ("C", "a"): + self.c = min(self.cols - 1, self.c + n()) + elif final == "D": + self.c = max(0, self.c - n()) + elif final == "E": + self.r, self.c = min(self.rows - 1, self.r + n()), 0 + elif final == "F": + self.r, self.c = max(0, self.r - n()), 0 + elif final in ("G", "`"): + self.c = min(self.cols - 1, n() - 1) + elif final == "d": + self.r = min(self.rows - 1, n() - 1) + elif final in ("H", "f"): + self.r = min(self.rows - 1, n(0) - 1) + self.c = min(self.cols - 1, n(1) - 1) + elif final == "J": + mode = nums[0] if nums else 0 + if mode == 0: + self._blank(self.r, self.c, self.cols) + for row in range(self.r + 1, self.rows): + self._blank(row, 0, self.cols) + elif mode == 1: + self._blank(self.r, 0, self.c + 1) + for row in range(0, self.r): + self._blank(row, 0, self.cols) + else: + for row in range(self.rows): + self._blank(row, 0, self.cols) + elif final == "K": + mode = nums[0] if nums else 0 + if mode == 0: + self._blank(self.r, self.c, self.cols) + elif mode == 1: + self._blank(self.r, 0, self.c + 1) + else: + self._blank(self.r, 0, self.cols) + elif final == "L": + for _ in range(n()): + self.grid.insert(self.r, [" "] * self.cols) + self.grid.pop() + elif final == "M": + for _ in range(n()): + self.grid.pop(self.r) + self.grid.append([" "] * self.cols) + elif final == "P": + count = min(n(), self.cols - self.c) + del self.grid[self.r][self.c:self.c + count] + self.grid[self.r] += [" "] * count + elif final == "@": + count = min(n(), self.cols - self.c) + self.grid[self.r][self.c:self.c] = [" "] * count + del self.grid[self.r][self.cols:] + elif final == "X": + self._blank(self.r, self.c, self.c + n()) + elif final == "S": + for _ in range(n()): + self.history.append("".join(self.grid.pop(0)).rstrip()) + self.grid.append([" "] * self.cols) + elif final == "T": + for _ in range(n()): + self.grid.insert(0, [" "] * self.cols) + self.grid.pop() + + +# An escape sequence still being typed out: ESC alone, a CSI or OSC whose terminator has not +# arrived, or a two-character sequence missing its second character. +INCOMPLETE_RE = re.compile(r"\x1b(?:\[[0-?]*[ -/]*|\][^\x07\x1b]*\x1b?|[ -/]*)?\Z") + + +def _incomplete_at_end(data: str) -> int | None: + m = INCOMPLETE_RE.search(data) + return m.start() if m else None + + +def _take_terminal() -> None: + fcntl.ioctl(0, termios.TIOCSCTTY, 0) + + +class PtySession(Session): + """A vendor's interactive UI on a pseudo-terminal: type a line, read the screen it draws.""" + + def __init__(self, argv: list[str], cwd: Path, env: dict, log_path: Path, + dialogs: tuple[tuple[str, str], ...] = (), reload_command: str | None = None, + exit_keys: tuple[str, ...] = ("\x03", "\x03", "\x04"), ready_idle: float = 4.0, + timeout: float = 180.0, cols: int = 200, rows: int = 50, submit_delay: float = 1.2) -> None: + self.argv = list(argv) + self.cwd = Path(cwd) + self.log_path = Path(log_path) + self.env = {**env, "TERM": "xterm-256color", "COLUMNS": str(cols), "LINES": str(rows)} + self.dialogs = [(re.compile(pattern), keys) for pattern, keys in dialogs] + self.reload_command = reload_command + self.exit_keys = tuple(exit_keys) + self.ready_idle = ready_idle + self.timeout = timeout + self.cols, self.rows = cols, rows + # A composer that opens as the text arrives swallows an Enter sent right behind it, and + # the prompt then sits unsent in a follow-up panel. + self.submit_delay = submit_delay + self.proc: subprocess.Popen | None = None + self.master = -1 + self.term = Terminal(rows, cols) + self._decoder = codecs.getincrementaldecoder("utf-8")("replace") + self._seen = False + self._lock = threading.Lock() + self._last = time.time() + self._answered: set[int] = set() + self.notes: list[str] = [] + self._noted = 0 + self.started_at = time.time() + + def start(self) -> None: + self.master, slave = pty.openpty() + fcntl.ioctl(self.master, termios.TIOCSWINSZ, struct.pack("HHHH", self.rows, self.cols, 0, 0)) + self._log = self.log_path.open("ab") + try: + # Its own session with the slave as controlling terminal, so the vendor sees a real + # tty and a stuck one can be killed with everything it spawned. + self.proc = subprocess.Popen(self.argv, cwd=str(self.cwd), env=self.env, stdin=slave, stdout=slave, + stderr=slave, start_new_session=True, preexec_fn=_take_terminal) + except BaseException: + # No child means close() has nothing to finish: release what was opened for it here. + os.close(self.master) + self._log.close() + raise + finally: + os.close(slave) + self._reader = threading.Thread(target=self._read, daemon=True) + self._reader.start() + try: + self.wait_ready(self.timeout) + except BaseException: + # A child that never became ready is still ours to stop: nobody holds this session. + self.close() + raise + + def _read(self) -> None: + while True: + try: + chunk = os.read(self.master, 65536) + except OSError: + break + if not chunk: + break + with self._lock: + self.term.feed(self._decoder.decode(chunk)) + self._seen = True + self._last = time.time() + self._log.write(chunk) + self._log.flush() + + def screen(self) -> str: + with self._lock: + return self.term.text() + + def send(self, text: str) -> None: + os.write(self.master, text.encode()) + + def _answer_dialogs(self) -> None: + # Never longer than the readiness window, or a vendor waiting on the dialog is declared + # ready and then typed at instead of answered. + settled = min(1.0, self.ready_idle) + with self._lock: + idle = time.time() - self._last + # A dialog answered while its list is still being drawn moves a selection that is about to + # be redrawn, and the Enter after it then confirms the default. + if idle < settled: + return + screen = self.screen() + for i, (pattern, keys) in enumerate(self.dialogs): + if i not in self._answered and pattern.search(screen): + self._answered.add(i) + self.notes.append(f"dialog={pattern.pattern}") + self.type(keys) + time.sleep(1.0) + + def type(self, keys: str, gap: float = 1.0) -> None: + """One keystroke per write, a redraw apart: an arrow and the Enter after it sent together + confirm the option the arrow was meant to leave.""" + for key in KEY_RE.findall(keys): + self.send(key) + time.sleep(gap) + + def wait_ready(self, timeout: float) -> None: + deadline = time.time() + timeout + while time.time() < deadline: + self._answer_dialogs() + with self._lock: + seen, idle = self._seen, time.time() - self._last + if seen and idle >= self.ready_idle: + return + if self.proc is not None and self.proc.poll() is not None: + raise RuntimeError(f"tui exited with {self.proc.returncode} before it was ready") + time.sleep(0.1) + raise TimeoutError(f"tui not ready within {timeout}s") + + def ask(self, prompt: str) -> AskResult: + # A wizard the vendor raises between turns would otherwise take the prompt as its input. + self._answer_dialogs() + # The screen starts blank for the turn, so an earlier turn's answer cannot be read as this + # one's; whatever the UI redraws afterwards is this turn's own output. + with self._lock: + self.term.clear() + self.send(prompt) + time.sleep(self.submit_delay) + first = time.time() + self.send("\r") + deadline = time.time() + self.timeout + reply = "" + extra: list[str] = [] + while time.time() < deadline: + self._answer_dialogs() + reply = extract_tui_reply(self.screen(), prompt) + if reply: + break + if self.proc is not None and self.proc.poll() is not None: + extra.append(f"tui exited with {self.proc.returncode}") + # The exit may follow the reply by less than one poll interval. + reply = extract_tui_reply(self.screen(), prompt) + break + time.sleep(0.2) + else: + extra.append("timeout") + if not reply: + extra.append("no PROBE-REPLY line on the screen") + # Dialog notes belong to the turn that answered them, once. + notes = self.notes[self._noted:] + extra + self._noted = len(self.notes) + return AskResult(reply_text=reply, raw=self.screen(), argv=list(self.argv), started_at=self.started_at, + first_request_at=first, stderr_path=str(self.log_path), exit_code=None, + notes=" ".join(notes)) + + def command(self, line: str, settle: float | None = None) -> str: + with self._lock: + self.term.clear() + # Enter goes in its own write: inside a burst a TUI's paste detection keeps it as text. + self.send(line) + time.sleep(self.submit_delay) + self.send("\r") + time.sleep(settle if settle is not None else max(self.ready_idle, 1.0)) + self._answer_dialogs() + return self.screen() + + def reload(self) -> str | None: + if self.reload_command is None: + return None + self.command(self.reload_command) + return self.reload_command + + def close(self) -> None: + if self.proc is None: + return + try: + for keys in self.exit_keys: + if self.proc.poll() is not None: + break + try: + self.send(keys) + except OSError: + break + time.sleep(0.5) + try: + self.proc.wait(timeout=5) + except subprocess.TimeoutExpired: + kill_group(self.proc.pid) + self.proc.wait(timeout=5) + finally: + kill_group(self.proc.pid) + try: + os.close(self.master) + except OSError: + pass + self._reader.join(timeout=5) + self._log.close() diff --git a/docs/probes/2026-09-16-skills-discovery/lib/recorder.py b/docs/probes/2026-09-16-skills-discovery/lib/recorder.py new file mode 100644 index 0000000000..0ca6f51f8b --- /dev/null +++ b/docs/probes/2026-09-16-skills-discovery/lib/recorder.py @@ -0,0 +1,107 @@ +from __future__ import annotations + +import json +import platform +import subprocess +from dataclasses import asdict, dataclass, field +from pathlib import Path + +from lib.verdict import combine + + +@dataclass +class RunRecord: + entry: str + harness: str + binary: str + version: str + os: str + mode: str + argv: list[str] + isolation_lever: str + credential_files: list[str] + auth_ok: bool | None + scenario: str + arm: str + root: str | None + exclusion: str + hook: dict | None + first_request_at: float | None + reply: str + tokens_found: list[str] + skill_named: bool + stderr_path: str | None + verdict: str + duration_ms: int + expected_tokens: dict[str, str] = field(default_factory=dict) + notes: str = "" + + +def os_label() -> str: + if platform.system() == "Darwin": + ver = subprocess.run(["sw_vers", "-productVersion"], capture_output=True, text=True).stdout.strip() + return f"macOS {ver} {platform.machine()}" + return f"{platform.system()} {platform.release()} {platform.machine()}" + + +def run_dir(outdir: Path, entry: str, mode: str, scenario: str, arm: str) -> Path: + return outdir / entry / mode / scenario / arm.replace("/", "_") + + +def next_run_path(outdir: Path, rec: RunRecord) -> Path: + d = run_dir(outdir, rec.entry, rec.mode, rec.scenario, rec.arm) + d.mkdir(parents=True, exist_ok=True) + n = 1 + while (d / f"run{n}.json").exists(): + n += 1 + return d / f"run{n}.json" + + +def write_run(outdir: Path, rec: RunRecord) -> Path: + path = next_run_path(outdir, rec) + path.write_text(json.dumps(asdict(rec), indent=2, sort_keys=True) + "\n") + return path + + +def load_runs(outdir: Path) -> list[RunRecord]: + runs = [] + for path in sorted(outdir.rglob("run*.json")): + data = json.loads(path.read_text()) + rec = RunRecord(**data) + rec._path = path # type: ignore[attr-defined] + runs.append(rec) + return runs + + +def _evidence(path: Path, target: Path) -> str: + try: + return str(path.relative_to(target.parent)) + except ValueError: + return str(path) + + +def emit_matrix(outdir: Path, target: Path) -> list[dict]: + runs = load_runs(outdir) + groups: dict[tuple, list[RunRecord]] = {} + for r in runs: + # Version is part of the key: runs from two versions of one entry are never averaged, + # and the older measurement keeps its own row rather than disappearing. + # The operating system is part of the key: two machines' runs are two findings, not an + # average of one. + groups.setdefault((r.entry, r.version, r.os, r.mode, r.scenario, r.arm, r.root or "", r.exclusion), + []).append(r) + rows = [] + for key in sorted(groups): + members = groups[key] + verdict, flaky = combine([m.verdict for m in members]) + first = members[0] + rows.append({ + "entry": first.entry, "harness": first.harness, "version": first.version, "os": first.os, + "mode": first.mode, "scenario": first.scenario, "arm": first.arm, "root": first.root, + "exclusion": first.exclusion, "verdict": verdict, "flaky": flaky, "runs": len(members), + "mechanism": (first.hook or {}).get("mechanism"), + "evidence": [_evidence(m._path, target) for m in members], # type: ignore[attr-defined] + "notes": "; ".join(dict.fromkeys(m.notes for m in members if m.notes)), + }) + target.write_text(json.dumps(rows, indent=2) + "\n") + return rows diff --git a/docs/probes/2026-09-16-skills-discovery/lib/verdict.py b/docs/probes/2026-09-16-skills-discovery/lib/verdict.py new file mode 100644 index 0000000000..da9beeda0b --- /dev/null +++ b/docs/probes/2026-09-16-skills-discovery/lib/verdict.py @@ -0,0 +1,70 @@ +from __future__ import annotations + +from collections import Counter + +from lib.probe_skill import Reply + +VERDICTS = ("visible_first_turn", "visible_after_reload", "visible_live", "catalogue_only", "not_visible", + "stale", "revoked", "leaked", "untested") + + +class PromptDesignFailure(Exception): + """The negative control produced a token: the prompt or parser is unsound for this entry.""" + + +def judge_single(expected_token: str, reply: Reply, reload_used: bool = False) -> str: + if expected_token in reply.tokens: + return "visible_after_reload" if reload_used else "visible_first_turn" + # Naming the skill while answering NO-SKILL is an echo of the prompt, not a sighting. + if reply.skill_named and not reply.no_skill: + return "catalogue_only" + return "not_visible" + + +def judge_control(reply: Reply) -> str: + if reply.tokens: + raise PromptDesignFailure(f"negative control produced tokens {sorted(reply.tokens)}") + return "not_visible" + + +def judge_root(token: str, documented: bool, reply: Reply) -> str: + if token in reply.tokens: + return "visible_first_turn" if documented else "leaked" + return "not_visible" + + +def combine(verdicts: list[str]) -> tuple[str, bool]: + if not verdicts: + return "untested", False + counts = Counter(verdicts) + top, _ = counts.most_common(1)[0] + return top, len(counts) > 1 + + +def needs_third_run(verdicts: list[str]) -> bool: + # A run that failed to measure is not a disagreement to break: the arm is cleared and + # measured again from scratch, and a third turn here would be spent on nothing. + if "untested" in verdicts: + return False + return len(verdicts) == 2 and verdicts[0] != verdicts[1] + + +def judge_update(new_token: str, old_token: str | None, reply: Reply, live: bool) -> str: + if new_token in reply.tokens: + return "visible_live" if live else "visible_first_turn" + if old_token is not None and old_token in reply.tokens: + return "stale" + if reply.skill_named and not reply.no_skill: + return "catalogue_only" + return "not_visible" + + +def judge_delete(old_token: str, reply: Reply) -> str: + if old_token in reply.tokens: + return "stale" + if reply.no_skill and not reply.tokens: + return "revoked" + # Still listed after the delete, even if unreadable: the catalogue did not follow the file. + if reply.skill_named: + return "stale" + return "not_visible" diff --git a/docs/probes/2026-09-16-skills-discovery/matrix.json b/docs/probes/2026-09-16-skills-discovery/matrix.json new file mode 100644 index 0000000000..76783eddf0 --- /dev/null +++ b/docs/probes/2026-09-16-skills-discovery/matrix.json @@ -0,0 +1,12691 @@ +[ + { + "entry": "agy", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S0", + "arm": "S0/none", + "root": null, + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/agy/print/S0/S0_none/run1.json", + "out/agy/print/S0/S0_none/run2.json" + ], + "notes": "slash=/kcap-probe-58df4b" + }, + { + "entry": "agy", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S1", + "arm": "S1/native", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/agy/print/S1/S1_native/run1.json", + "out/agy/print/S1/S1_native/run2.json" + ], + "notes": "slash=/kcap-probe-929e9c; slash=/kcap-probe-f1b555" + }, + { + "entry": "agy", + "harness": "antigravity", + "version": "1.2.5", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S2", + "arm": "S2/hook-adds-skill", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/agy/print/S2/S2_hook-adds-skill/run1.json" + ], + "notes": "S1 failed" + }, + { + "entry": "agy", + "harness": "antigravity", + "version": "1.2.5", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S2", + "arm": "S2/hook-creates-root", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/agy/print/S2/S2_hook-creates-root/run1.json" + ], + "notes": "S1 failed" + }, + { + "entry": "agy", + "harness": "antigravity", + "version": "1.2.5", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S2", + "arm": "S2/registration", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/agy/print/S2/S2_registration/run1.json" + ], + "notes": "S1 failed" + }, + { + "entry": "agy", + "harness": "antigravity", + "version": "1.2.5", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S3", + "arm": "S3/gitignore", + "root": ".agents/skills", + "exclusion": "gitignore", + "verdict": "untested", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/agy/print/S3/S3_gitignore/run1.json" + ], + "notes": "S1 failed" + }, + { + "entry": "agy", + "harness": "antigravity", + "version": "1.2.5", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S3", + "arm": "S3/info-exclude", + "root": ".agents/skills", + "exclusion": "info-exclude", + "verdict": "untested", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/agy/print/S3/S3_info-exclude/run1.json" + ], + "notes": "S1 failed" + }, + { + "entry": "agy", + "harness": "antigravity", + "version": "1.2.5", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": null, + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/agy/print/S4/S4_all-roots/run1.json" + ], + "notes": "S1 failed" + }, + { + "entry": "agy-clidir", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S0", + "arm": "S0/none", + "root": null, + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/agy-clidir/print/S0/S0_none/run1.json", + "out/agy-clidir/print/S0/S0_none/run2.json" + ], + "notes": "slash=/kcap-probe-db9de1; slash=/kcap-probe-8156d7" + }, + { + "entry": "agy-clidir", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S1", + "arm": "S1/native", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/agy-clidir/print/S1/S1_native/run1.json", + "out/agy-clidir/print/S1/S1_native/run2.json" + ], + "notes": "slash=/kcap-probe-842287; slash=/kcap-probe-eefbb5" + }, + { + "entry": "agy-clidir", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S2", + "arm": "S2/hook-adds-skill", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 2, + "mechanism": "~/.gemini/antigravity-cli/plugins PreInvocation", + "evidence": [ + "out/agy-clidir/print/S2/S2_hook-adds-skill/run1.json", + "out/agy-clidir/print/S2/S2_hook-adds-skill/run2.json" + ], + "notes": "hook never fired skill file absent after the turn slash=/kcap-probe-fa85ff; hook never fired skill file absent after the turn slash=/kcap-probe-84de58" + }, + { + "entry": "agy-clidir", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S2", + "arm": "S2/hook-creates-root", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 2, + "mechanism": "~/.gemini/antigravity-cli/plugins PreInvocation", + "evidence": [ + "out/agy-clidir/print/S2/S2_hook-creates-root/run1.json", + "out/agy-clidir/print/S2/S2_hook-creates-root/run2.json" + ], + "notes": "hook never fired skill file absent after the turn slash=/kcap-probe-207740; hook never fired skill file absent after the turn slash=/kcap-probe-d6ffe0" + }, + { + "entry": "agy-clidir", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S2", + "arm": "S2/registration", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/agy-clidir/print/S2/S2_registration/run1.json" + ], + "notes": "no registration mechanism for this entry" + }, + { + "entry": "agy-clidir", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S3", + "arm": "S3/gitignore", + "root": ".agents/skills", + "exclusion": "gitignore", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/agy-clidir/print/S3/S3_gitignore/run1.json", + "out/agy-clidir/print/S3/S3_gitignore/run2.json" + ], + "notes": "slash=/kcap-probe-0037cf; slash=/kcap-probe-af8f98" + }, + { + "entry": "agy-clidir", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S3", + "arm": "S3/info-exclude", + "root": ".agents/skills", + "exclusion": "info-exclude", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/agy-clidir/print/S3/S3_info-exclude/run1.json", + "out/agy-clidir/print/S3/S3_info-exclude/run2.json" + ], + "notes": "slash=/kcap-probe-630e0b; slash=/kcap-probe-8df12d" + }, + { + "entry": "agy-clidir", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".agent/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/agy-clidir/print/S4/S4_all-roots/run10.json", + "out/agy-clidir/print/S4/S4_all-roots/run20.json" + ], + "notes": "found=['agent', 'agents'] leaked=[]" + }, + { + "entry": "agy-clidir", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/agy-clidir/print/S4/S4_all-roots/run12.json", + "out/agy-clidir/print/S4/S4_all-roots/run2.json" + ], + "notes": "found=['agent', 'agents'] leaked=[]" + }, + { + "entry": "agy-clidir", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".claude/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/agy-clidir/print/S4/S4_all-roots/run1.json", + "out/agy-clidir/print/S4/S4_all-roots/run11.json" + ], + "notes": "found=['agent', 'agents'] leaked=[]" + }, + { + "entry": "agy-clidir", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".codex/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/agy-clidir/print/S4/S4_all-roots/run13.json", + "out/agy-clidir/print/S4/S4_all-roots/run3.json" + ], + "notes": "found=['agent', 'agents'] leaked=[]" + }, + { + "entry": "agy-clidir", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/agy-clidir/print/S4/S4_all-roots/run14.json", + "out/agy-clidir/print/S4/S4_all-roots/run4.json" + ], + "notes": "found=['agent', 'agents'] leaked=[]" + }, + { + "entry": "agy-clidir", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/agy-clidir/print/S4/S4_all-roots/run16.json", + "out/agy-clidir/print/S4/S4_all-roots/run6.json" + ], + "notes": "found=['agent', 'agents'] leaked=[]" + }, + { + "entry": "agy-clidir", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".github/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/agy-clidir/print/S4/S4_all-roots/run15.json", + "out/agy-clidir/print/S4/S4_all-roots/run5.json" + ], + "notes": "found=['agent', 'agents'] leaked=[]" + }, + { + "entry": "agy-clidir", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/agy-clidir/print/S4/S4_all-roots/run17.json", + "out/agy-clidir/print/S4/S4_all-roots/run7.json" + ], + "notes": "found=['agent', 'agents'] leaked=[]" + }, + { + "entry": "agy-clidir", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/agy-clidir/print/S4/S4_all-roots/run19.json", + "out/agy-clidir/print/S4/S4_all-roots/run9.json" + ], + "notes": "found=['agent', 'agents'] leaked=[]" + }, + { + "entry": "agy-clidir", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/agy-clidir/print/S4/S4_all-roots/run18.json", + "out/agy-clidir/print/S4/S4_all-roots/run8.json" + ], + "notes": "found=['agent', 'agents'] leaked=[]" + }, + { + "entry": "agy-clidir", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-claude", + "root": ".claude/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/agy-clidir/print/S4/S4_confirm-claude/run1.json" + ], + "notes": "slash=/kcap-probe-456006" + }, + { + "entry": "agy-clidir", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-codex", + "root": ".codex/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/agy-clidir/print/S4/S4_confirm-codex/run1.json" + ], + "notes": "slash=/kcap-probe-f8199f" + }, + { + "entry": "agy-clidir", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-cursor", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/agy-clidir/print/S4/S4_confirm-cursor/run1.json" + ], + "notes": "slash=/kcap-probe-3a0644" + }, + { + "entry": "agy-clidir", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-gemini", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/agy-clidir/print/S4/S4_confirm-gemini/run1.json" + ], + "notes": "slash=/kcap-probe-61136d" + }, + { + "entry": "agy-clidir", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-github", + "root": ".github/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/agy-clidir/print/S4/S4_confirm-github/run1.json" + ], + "notes": "slash=/kcap-probe-1c3a57" + }, + { + "entry": "agy-clidir", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-kiro", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/agy-clidir/print/S4/S4_confirm-kiro/run1.json" + ], + "notes": "slash=/kcap-probe-1ebd45" + }, + { + "entry": "agy-clidir", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-opencode", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/agy-clidir/print/S4/S4_confirm-opencode/run1.json" + ], + "notes": "slash=/kcap-probe-6d1857" + }, + { + "entry": "agy-clidir", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-pi", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/agy-clidir/print/S4/S4_confirm-pi/run1.json" + ], + "notes": "slash=/kcap-probe-0d46d0" + }, + { + "entry": "agy-dirlayout", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S0", + "arm": "S0/none", + "root": null, + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/agy-dirlayout/print/S0/S0_none/run1.json", + "out/agy-dirlayout/print/S0/S0_none/run2.json" + ], + "notes": "slash=/kcap-probe-598b65; slash=/kcap-probe-f000c0" + }, + { + "entry": "agy-dirlayout", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S1", + "arm": "S1/native", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/agy-dirlayout/print/S1/S1_native/run1.json", + "out/agy-dirlayout/print/S1/S1_native/run2.json" + ], + "notes": "slash=/kcap-probe-051be9; slash=/kcap-probe-1f8282" + }, + { + "entry": "agy-dirlayout", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S2", + "arm": "S2/hook-adds-skill", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": "workspace .agents/plugins PreInvocation", + "evidence": [ + "out/agy-dirlayout/print/S2/S2_hook-adds-skill/run1.json", + "out/agy-dirlayout/print/S2/S2_hook-adds-skill/run2.json" + ], + "notes": "slash=/kcap-probe-9d4a41; slash=/kcap-probe-4d6f9e" + }, + { + "entry": "agy-dirlayout", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S2", + "arm": "S2/hook-creates-root", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": "workspace .agents/plugins PreInvocation", + "evidence": [ + "out/agy-dirlayout/print/S2/S2_hook-creates-root/run1.json", + "out/agy-dirlayout/print/S2/S2_hook-creates-root/run2.json" + ], + "notes": "slash=/kcap-probe-22702b; slash=/kcap-probe-bf4489" + }, + { + "entry": "agy-dirlayout", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S2", + "arm": "S2/registration", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/agy-dirlayout/print/S2/S2_registration/run1.json" + ], + "notes": "no registration mechanism for this entry" + }, + { + "entry": "agy-dirlayout", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S3", + "arm": "S3/gitignore", + "root": ".agents/skills", + "exclusion": "gitignore", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/agy-dirlayout/print/S3/S3_gitignore/run1.json", + "out/agy-dirlayout/print/S3/S3_gitignore/run2.json" + ], + "notes": "slash=/kcap-probe-9d6f7a; slash=/kcap-probe-84fb44" + }, + { + "entry": "agy-dirlayout", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S3", + "arm": "S3/info-exclude", + "root": ".agents/skills", + "exclusion": "info-exclude", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/agy-dirlayout/print/S3/S3_info-exclude/run1.json", + "out/agy-dirlayout/print/S3/S3_info-exclude/run2.json" + ], + "notes": "slash=/kcap-probe-740721; slash=/kcap-probe-08f5c0" + }, + { + "entry": "agy-dirlayout", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".agent/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/agy-dirlayout/print/S4/S4_all-roots/run10.json", + "out/agy-dirlayout/print/S4/S4_all-roots/run20.json" + ], + "notes": "found=['agent', 'agents'] leaked=[]" + }, + { + "entry": "agy-dirlayout", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/agy-dirlayout/print/S4/S4_all-roots/run12.json", + "out/agy-dirlayout/print/S4/S4_all-roots/run2.json" + ], + "notes": "found=['agent', 'agents'] leaked=[]" + }, + { + "entry": "agy-dirlayout", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".claude/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/agy-dirlayout/print/S4/S4_all-roots/run1.json", + "out/agy-dirlayout/print/S4/S4_all-roots/run11.json" + ], + "notes": "found=['agent', 'agents'] leaked=[]" + }, + { + "entry": "agy-dirlayout", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".codex/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/agy-dirlayout/print/S4/S4_all-roots/run13.json", + "out/agy-dirlayout/print/S4/S4_all-roots/run3.json" + ], + "notes": "found=['agent', 'agents'] leaked=[]" + }, + { + "entry": "agy-dirlayout", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/agy-dirlayout/print/S4/S4_all-roots/run14.json", + "out/agy-dirlayout/print/S4/S4_all-roots/run4.json" + ], + "notes": "found=['agent', 'agents'] leaked=[]" + }, + { + "entry": "agy-dirlayout", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/agy-dirlayout/print/S4/S4_all-roots/run16.json", + "out/agy-dirlayout/print/S4/S4_all-roots/run6.json" + ], + "notes": "found=['agent', 'agents'] leaked=[]" + }, + { + "entry": "agy-dirlayout", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".github/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/agy-dirlayout/print/S4/S4_all-roots/run15.json", + "out/agy-dirlayout/print/S4/S4_all-roots/run5.json" + ], + "notes": "found=['agent', 'agents'] leaked=[]" + }, + { + "entry": "agy-dirlayout", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/agy-dirlayout/print/S4/S4_all-roots/run17.json", + "out/agy-dirlayout/print/S4/S4_all-roots/run7.json" + ], + "notes": "found=['agent', 'agents'] leaked=[]" + }, + { + "entry": "agy-dirlayout", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/agy-dirlayout/print/S4/S4_all-roots/run19.json", + "out/agy-dirlayout/print/S4/S4_all-roots/run9.json" + ], + "notes": "found=['agent', 'agents'] leaked=[]" + }, + { + "entry": "agy-dirlayout", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/agy-dirlayout/print/S4/S4_all-roots/run18.json", + "out/agy-dirlayout/print/S4/S4_all-roots/run8.json" + ], + "notes": "found=['agent', 'agents'] leaked=[]" + }, + { + "entry": "agy-dirlayout", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-claude", + "root": ".claude/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/agy-dirlayout/print/S4/S4_confirm-claude/run1.json" + ], + "notes": "slash=/kcap-probe-032124" + }, + { + "entry": "agy-dirlayout", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-codex", + "root": ".codex/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/agy-dirlayout/print/S4/S4_confirm-codex/run1.json" + ], + "notes": "slash=/kcap-probe-830422" + }, + { + "entry": "agy-dirlayout", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-cursor", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/agy-dirlayout/print/S4/S4_confirm-cursor/run1.json" + ], + "notes": "slash=/kcap-probe-ba67ba" + }, + { + "entry": "agy-dirlayout", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-gemini", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/agy-dirlayout/print/S4/S4_confirm-gemini/run1.json" + ], + "notes": "slash=/kcap-probe-f0b350" + }, + { + "entry": "agy-dirlayout", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-github", + "root": ".github/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/agy-dirlayout/print/S4/S4_confirm-github/run1.json" + ], + "notes": "slash=/kcap-probe-521bf6" + }, + { + "entry": "agy-dirlayout", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-kiro", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/agy-dirlayout/print/S4/S4_confirm-kiro/run1.json" + ], + "notes": "slash=/kcap-probe-8472e2" + }, + { + "entry": "agy-dirlayout", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-opencode", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/agy-dirlayout/print/S4/S4_confirm-opencode/run1.json" + ], + "notes": "slash=/kcap-probe-c1ebe8" + }, + { + "entry": "agy-dirlayout", + "harness": "antigravity", + "version": "1.2.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-pi", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/agy-dirlayout/print/S4/S4_confirm-pi/run1.json" + ], + "notes": "slash=/kcap-probe-7fa958" + }, + { + "entry": "agy-dirlayout", + "harness": "antigravity", + "version": "1.2.5", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S6", + "arm": "S6/delete", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "stale", + "flaky": false, + "runs": 2, + "mechanism": "workspace .agents/plugins PreInvocation", + "evidence": [ + "out/agy-dirlayout/print/S6/S6_delete/run1.json", + "out/agy-dirlayout/print/S6/S6_delete/run2.json" + ], + "notes": "slash=/kcap-probe-26a1dc; slash=/kcap-probe-ce8ec3" + }, + { + "entry": "agy-dirlayout", + "harness": "antigravity", + "version": "1.2.5", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S6", + "arm": "S6/update", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": "workspace .agents/plugins PreInvocation", + "evidence": [ + "out/agy-dirlayout/print/S6/S6_update/run1.json", + "out/agy-dirlayout/print/S6/S6_update/run2.json" + ], + "notes": "slash=/kcap-probe-de09bf; slash=/kcap-probe-4d51fd" + }, + { + "entry": "agy-dirlayout", + "harness": "antigravity", + "version": "1.2.5", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S7", + "arm": "S7/add", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/agy-dirlayout/print/S7/S7_add/run1.json", + "out/agy-dirlayout/print/S7/S7_add/run2.json" + ], + "notes": "turn1=not_visible session=6b421af4-6603-430a-979d-30a9ec866f21 slash=/kcap-probe-3253a7; turn1=not_visible session=d5e756df-4f5d-4c84-a72c-6f7d20d823e4 slash=/kcap-probe-a70618" + }, + { + "entry": "agy-dirlayout", + "harness": "antigravity", + "version": "1.2.5", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S7", + "arm": "S7/update", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/agy-dirlayout/print/S7/S7_update/run1.json", + "out/agy-dirlayout/print/S7/S7_update/run2.json" + ], + "notes": "turn1=visible_first_turn session=ff6a0311-586f-4dd8-a5bd-fc7c869e395b slash=/kcap-probe-78bf18; turn1=visible_first_turn session=ba2693f7-d3c5-4b27-8ea0-e1a74091cf7c slash=/kcap-probe-f1a4df" + }, + { + "entry": "agy-dirlayout", + "harness": "antigravity", + "version": "1.2.5", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S8", + "arm": "S8/ancestor", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/agy-dirlayout/print/S8/S8_ancestor/run1.json", + "out/agy-dirlayout/print/S8/S8_ancestor/run2.json" + ], + "notes": "cwd=sub/dir slash=/kcap-probe-b01637; cwd=sub/dir slash=/kcap-probe-94ad17" + }, + { + "entry": "agy-dirlayout", + "harness": "antigravity", + "version": "1.2.5", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S8", + "arm": "S8/local", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/agy-dirlayout/print/S8/S8_local/run1.json", + "out/agy-dirlayout/print/S8/S8_local/run2.json" + ], + "notes": "cwd=sub/dir slash=/kcap-probe-f7c966; cwd=sub/dir slash=/kcap-probe-f521f9" + }, + { + "entry": "agy-dirlayout", + "harness": "antigravity", + "version": "1.2.5", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S9", + "arm": "S9/linked-other", + "root": ".agents/skills", + "exclusion": "info-exclude", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/agy-dirlayout/print/S9/S9_linked-other/run1.json", + "out/agy-dirlayout/print/S9/S9_linked-other/run2.json" + ], + "notes": "cwd=wt-b skill_in=main slash=/kcap-probe-546a67; cwd=wt-b skill_in=main slash=/kcap-probe-586864" + }, + { + "entry": "agy-dirlayout", + "harness": "antigravity", + "version": "1.2.5", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S9", + "arm": "S9/linked-own", + "root": ".agents/skills", + "exclusion": "info-exclude", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/agy-dirlayout/print/S9/S9_linked-own/run1.json", + "out/agy-dirlayout/print/S9/S9_linked-own/run2.json" + ], + "notes": "cwd=wt-b skill_in=linked slash=/kcap-probe-ec31ab; cwd=wt-b skill_in=linked slash=/kcap-probe-0eb1a2" + }, + { + "entry": "agy-dirlayout", + "harness": "antigravity", + "version": "1.2.5", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S0", + "arm": "S0/none", + "root": null, + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/agy-dirlayout/tui/S0/S0_none/run1.json", + "out/agy-dirlayout/tui/S0/S0_none/run2.json" + ], + "notes": "" + }, + { + "entry": "agy-dirlayout", + "harness": "antigravity", + "version": "1.2.5", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S1", + "arm": "S1/native", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/agy-dirlayout/tui/S1/S1_native/run1.json", + "out/agy-dirlayout/tui/S1/S1_native/run2.json" + ], + "notes": "" + }, + { + "entry": "agy-dirlayout", + "harness": "antigravity", + "version": "1.2.5", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S2", + "arm": "S2/hook-adds-skill", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": "workspace .agents/plugins PreInvocation", + "evidence": [ + "out/agy-dirlayout/tui/S2/S2_hook-adds-skill/run1.json", + "out/agy-dirlayout/tui/S2/S2_hook-adds-skill/run2.json" + ], + "notes": "" + }, + { + "entry": "agy-dirlayout", + "harness": "antigravity", + "version": "1.2.5", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S5", + "arm": "S5/add", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "visible_live", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/agy-dirlayout/tui/S5/S5_add/run1.json", + "out/agy-dirlayout/tui/S5/S5_add/run2.json" + ], + "notes": "turn1=not_visible" + }, + { + "entry": "agy-dirlayout", + "harness": "antigravity", + "version": "1.2.5", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S5", + "arm": "S5/reload", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/agy-dirlayout/tui/S5/S5_reload/run1.json" + ], + "notes": "no reload command for this entry" + }, + { + "entry": "claude", + "harness": "claude", + "version": "2.1.273 (Claude Code)", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S0", + "arm": "S0/none", + "root": null, + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/claude/print/S0/S0_none/run1.json", + "out/claude/print/S0/S0_none/run2.json" + ], + "notes": "config=real" + }, + { + "entry": "claude", + "harness": "claude", + "version": "2.1.273 (Claude Code)", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S1", + "arm": "S1/native", + "root": ".claude/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/claude/print/S1/S1_native/run1.json", + "out/claude/print/S1/S1_native/run2.json" + ], + "notes": "config=real" + }, + { + "entry": "claude", + "harness": "claude", + "version": "2.1.273 (Claude Code)", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S2", + "arm": "S2/hook-adds-skill", + "root": ".claude/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": "--settings file over the real config root hooks.SessionStart", + "evidence": [ + "out/claude/print/S2/S2_hook-adds-skill/run1.json", + "out/claude/print/S2/S2_hook-adds-skill/run2.json" + ], + "notes": "config=real" + }, + { + "entry": "claude", + "harness": "claude", + "version": "2.1.273 (Claude Code)", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S2", + "arm": "S2/hook-creates-root", + "root": ".claude/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": "--settings file over the real config root hooks.SessionStart", + "evidence": [ + "out/claude/print/S2/S2_hook-creates-root/run1.json", + "out/claude/print/S2/S2_hook-creates-root/run2.json" + ], + "notes": "config=real" + }, + { + "entry": "claude", + "harness": "claude", + "version": "2.1.273 (Claude Code)", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S2", + "arm": "S2/registration", + "root": ".claude/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/claude/print/S2/S2_registration/run1.json" + ], + "notes": "no registration mechanism for this entry" + }, + { + "entry": "claude", + "harness": "claude", + "version": "2.1.273 (Claude Code)", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S3", + "arm": "S3/gitignore", + "root": ".claude/skills", + "exclusion": "gitignore", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/claude/print/S3/S3_gitignore/run1.json", + "out/claude/print/S3/S3_gitignore/run2.json" + ], + "notes": "config=real" + }, + { + "entry": "claude", + "harness": "claude", + "version": "2.1.273 (Claude Code)", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S3", + "arm": "S3/info-exclude", + "root": ".claude/skills", + "exclusion": "info-exclude", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/claude/print/S3/S3_info-exclude/run1.json", + "out/claude/print/S3/S3_info-exclude/run2.json" + ], + "notes": "config=real" + }, + { + "entry": "claude", + "harness": "claude", + "version": "2.1.273 (Claude Code)", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".agent/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/claude/print/S4/S4_all-roots/run10.json", + "out/claude/print/S4/S4_all-roots/run20.json" + ], + "notes": "found=['claude'] leaked=[] config=real" + }, + { + "entry": "claude", + "harness": "claude", + "version": "2.1.273 (Claude Code)", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/claude/print/S4/S4_all-roots/run12.json", + "out/claude/print/S4/S4_all-roots/run2.json" + ], + "notes": "found=['claude'] leaked=[] config=real" + }, + { + "entry": "claude", + "harness": "claude", + "version": "2.1.273 (Claude Code)", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".claude/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/claude/print/S4/S4_all-roots/run1.json", + "out/claude/print/S4/S4_all-roots/run11.json" + ], + "notes": "found=['claude'] leaked=[] config=real" + }, + { + "entry": "claude", + "harness": "claude", + "version": "2.1.273 (Claude Code)", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".codex/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/claude/print/S4/S4_all-roots/run13.json", + "out/claude/print/S4/S4_all-roots/run3.json" + ], + "notes": "found=['claude'] leaked=[] config=real" + }, + { + "entry": "claude", + "harness": "claude", + "version": "2.1.273 (Claude Code)", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/claude/print/S4/S4_all-roots/run14.json", + "out/claude/print/S4/S4_all-roots/run4.json" + ], + "notes": "found=['claude'] leaked=[] config=real" + }, + { + "entry": "claude", + "harness": "claude", + "version": "2.1.273 (Claude Code)", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/claude/print/S4/S4_all-roots/run16.json", + "out/claude/print/S4/S4_all-roots/run6.json" + ], + "notes": "found=['claude'] leaked=[] config=real" + }, + { + "entry": "claude", + "harness": "claude", + "version": "2.1.273 (Claude Code)", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".github/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/claude/print/S4/S4_all-roots/run15.json", + "out/claude/print/S4/S4_all-roots/run5.json" + ], + "notes": "found=['claude'] leaked=[] config=real" + }, + { + "entry": "claude", + "harness": "claude", + "version": "2.1.273 (Claude Code)", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/claude/print/S4/S4_all-roots/run17.json", + "out/claude/print/S4/S4_all-roots/run7.json" + ], + "notes": "found=['claude'] leaked=[] config=real" + }, + { + "entry": "claude", + "harness": "claude", + "version": "2.1.273 (Claude Code)", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/claude/print/S4/S4_all-roots/run19.json", + "out/claude/print/S4/S4_all-roots/run9.json" + ], + "notes": "found=['claude'] leaked=[] config=real" + }, + { + "entry": "claude", + "harness": "claude", + "version": "2.1.273 (Claude Code)", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/claude/print/S4/S4_all-roots/run18.json", + "out/claude/print/S4/S4_all-roots/run8.json" + ], + "notes": "found=['claude'] leaked=[] config=real" + }, + { + "entry": "claude", + "harness": "claude", + "version": "2.1.273 (Claude Code)", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-agent", + "root": ".agent/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/claude/print/S4/S4_confirm-agent/run1.json" + ], + "notes": "config=real" + }, + { + "entry": "claude", + "harness": "claude", + "version": "2.1.273 (Claude Code)", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-agents", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/claude/print/S4/S4_confirm-agents/run1.json" + ], + "notes": "config=real" + }, + { + "entry": "claude", + "harness": "claude", + "version": "2.1.273 (Claude Code)", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-codex", + "root": ".codex/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/claude/print/S4/S4_confirm-codex/run1.json" + ], + "notes": "config=real" + }, + { + "entry": "claude", + "harness": "claude", + "version": "2.1.273 (Claude Code)", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-cursor", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/claude/print/S4/S4_confirm-cursor/run1.json" + ], + "notes": "config=real" + }, + { + "entry": "claude", + "harness": "claude", + "version": "2.1.273 (Claude Code)", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-gemini", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/claude/print/S4/S4_confirm-gemini/run1.json" + ], + "notes": "config=real" + }, + { + "entry": "claude", + "harness": "claude", + "version": "2.1.273 (Claude Code)", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-github", + "root": ".github/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/claude/print/S4/S4_confirm-github/run1.json" + ], + "notes": "config=real" + }, + { + "entry": "claude", + "harness": "claude", + "version": "2.1.273 (Claude Code)", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-kiro", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/claude/print/S4/S4_confirm-kiro/run1.json" + ], + "notes": "config=real" + }, + { + "entry": "claude", + "harness": "claude", + "version": "2.1.273 (Claude Code)", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-opencode", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/claude/print/S4/S4_confirm-opencode/run1.json" + ], + "notes": "config=real" + }, + { + "entry": "claude", + "harness": "claude", + "version": "2.1.273 (Claude Code)", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-pi", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/claude/print/S4/S4_confirm-pi/run1.json" + ], + "notes": "config=real" + }, + { + "entry": "claude", + "harness": "claude", + "version": "2.1.274 (Claude Code)", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S6", + "arm": "S6/delete", + "root": ".claude/skills", + "exclusion": "none", + "verdict": "revoked", + "flaky": false, + "runs": 2, + "mechanism": "--settings file over the real config root hooks.SessionStart", + "evidence": [ + "out/claude/print/S6/S6_delete/run1.json", + "out/claude/print/S6/S6_delete/run2.json" + ], + "notes": "config=real" + }, + { + "entry": "claude", + "harness": "claude", + "version": "2.1.274 (Claude Code)", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S6", + "arm": "S6/update", + "root": ".claude/skills", + "exclusion": "none", + "verdict": "stale", + "flaky": false, + "runs": 2, + "mechanism": "--settings file over the real config root hooks.SessionStart", + "evidence": [ + "out/claude/print/S6/S6_update/run1.json", + "out/claude/print/S6/S6_update/run2.json" + ], + "notes": "config=real" + }, + { + "entry": "claude", + "harness": "claude", + "version": "2.1.274 (Claude Code)", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S7", + "arm": "S7/add", + "root": ".claude/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/claude/print/S7/S7_add/run1.json", + "out/claude/print/S7/S7_add/run2.json" + ], + "notes": "turn1=not_visible session=d20a511d-9886-4f8a-b333-af10bcd8704b config=real; turn1=not_visible session=ab307e85-08a5-426c-a266-4c4aba45a125 config=real" + }, + { + "entry": "claude", + "harness": "claude", + "version": "2.1.274 (Claude Code)", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S7", + "arm": "S7/update", + "root": ".claude/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/claude/print/S7/S7_update/run1.json", + "out/claude/print/S7/S7_update/run2.json" + ], + "notes": "exception=RuntimeError('no reply read from the output; exit 1; extract failed: ValueError(\\'is_error: {\"duration_api_ms\": 671, \"stop_reason\": \"refusal\", \"session_id\": \"018f4770-b630-411c-ba1d-64e08841cabe\", \"total_cost_usd\": 0.010109, \"usage\": {\"input_tokens\": 28, \"cache_creation_input_tokens\": 148, \"cache_read_input_tokens\": 16978, \"output_tokens\": 0, \"output_tokens_details\": {\"thinking_tokens\": 0}\\') config=real; stderr: '); exception=RuntimeError('no reply read from the output; exit 1; extract failed: ValueError(\\'is_error: {\"duration_api_ms\": 694, \"stop_reason\": \"refusal\", \"session_id\": \"ce011733-eb16-47b2-b24f-5a7d452250a2\", \"total_cost_usd\": 0.010121999999999999, \"usage\": {\"input_tokens\": 28, \"cache_creation_input_tokens\": 149, \"cache_read_input_tokens\": 16984, \"output_tokens\": 0, \"output_tokens_details\": {\"thinking\\') config=real; stderr: ')" + }, + { + "entry": "claude", + "harness": "claude", + "version": "2.1.274 (Claude Code)", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S8", + "arm": "S8/ancestor", + "root": ".claude/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/claude/print/S8/S8_ancestor/run1.json", + "out/claude/print/S8/S8_ancestor/run2.json" + ], + "notes": "cwd=sub/dir config=real" + }, + { + "entry": "claude", + "harness": "claude", + "version": "2.1.274 (Claude Code)", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S8", + "arm": "S8/local", + "root": ".claude/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/claude/print/S8/S8_local/run1.json", + "out/claude/print/S8/S8_local/run2.json" + ], + "notes": "cwd=sub/dir config=real" + }, + { + "entry": "claude", + "harness": "claude", + "version": "2.1.274 (Claude Code)", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S9", + "arm": "S9/linked-other", + "root": ".claude/skills", + "exclusion": "info-exclude", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/claude/print/S9/S9_linked-other/run1.json", + "out/claude/print/S9/S9_linked-other/run2.json" + ], + "notes": "cwd=wt-b skill_in=main config=real" + }, + { + "entry": "claude", + "harness": "claude", + "version": "2.1.274 (Claude Code)", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S9", + "arm": "S9/linked-own", + "root": ".claude/skills", + "exclusion": "info-exclude", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/claude/print/S9/S9_linked-own/run1.json", + "out/claude/print/S9/S9_linked-own/run2.json" + ], + "notes": "cwd=wt-b skill_in=linked config=real" + }, + { + "entry": "claude", + "harness": "claude", + "version": "2.1.274 (Claude Code)", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S0", + "arm": "S0/none", + "root": null, + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/claude/tui/S0/S0_none/run1.json", + "out/claude/tui/S0/S0_none/run2.json" + ], + "notes": "dialog=Yes, I trust this folder" + }, + { + "entry": "claude", + "harness": "claude", + "version": "2.1.274 (Claude Code)", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S1", + "arm": "S1/native", + "root": ".claude/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/claude/tui/S1/S1_native/run1.json", + "out/claude/tui/S1/S1_native/run2.json" + ], + "notes": "dialog=Yes, I trust this folder" + }, + { + "entry": "claude", + "harness": "claude", + "version": "2.1.274 (Claude Code)", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S2", + "arm": "S2/hook-adds-skill", + "root": ".claude/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": "--settings file over the real config root hooks.SessionStart", + "evidence": [ + "out/claude/tui/S2/S2_hook-adds-skill/run1.json", + "out/claude/tui/S2/S2_hook-adds-skill/run2.json" + ], + "notes": "dialog=Yes, I trust this folder" + }, + { + "entry": "claude", + "harness": "claude", + "version": "2.1.274 (Claude Code)", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S5", + "arm": "S5/add", + "root": ".claude/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/claude/tui/S5/S5_add/run1.json", + "out/claude/tui/S5/S5_add/run2.json" + ], + "notes": "turn1=not_visible" + }, + { + "entry": "claude", + "harness": "claude", + "version": "2.1.274 (Claude Code)", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S5", + "arm": "S5/reload", + "root": ".claude/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/claude/tui/S5/S5_reload/run1.json" + ], + "notes": "no reload command for this entry" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S0", + "arm": "S0/none", + "root": null, + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/codex/daemon/S0/S0_none/run1.json", + "out/codex/daemon/S0/S0_none/run2.json" + ], + "notes": "hook_trust=none turn=completed tools_used=0 tools_used=0 skill_reads=0 searches=0; hook_trust=none turn=completed tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S1", + "arm": "S1/native", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/codex/daemon/S1/S1_native/run1.json", + "out/codex/daemon/S1/S1_native/run2.json" + ], + "notes": "hook_trust=none turn=completed tools_used=1 tools_used=0 skill_reads=1 searches=0; hook_trust=none turn=completed tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S10", + "arm": "S10/hook-from-peer", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "visible_live", + "flaky": false, + "runs": 2, + "mechanism": "hooks.json SessionStart", + "evidence": [ + "out/codex/daemon/S10/S10_hook-from-peer/run1.json", + "out/codex/daemon/S10/S10_hook-from-peer/run2.json" + ], + "notes": "turn1=not_visible peer=not_visible hook_trust=none turn=completed tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S2", + "arm": "S2/hook-adds-skill", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": "hooks.json SessionStart", + "evidence": [ + "out/codex/daemon/S2/S2_hook-adds-skill/run1.json", + "out/codex/daemon/S2/S2_hook-adds-skill/run2.json" + ], + "notes": "hook_trust=seeded turn=completed tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S2", + "arm": "S2/hook-creates-root", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": "hooks.json SessionStart", + "evidence": [ + "out/codex/daemon/S2/S2_hook-creates-root/run1.json", + "out/codex/daemon/S2/S2_hook-creates-root/run2.json" + ], + "notes": "hook_trust=seeded turn=completed tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S2", + "arm": "S2/registration", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/codex/daemon/S2/S2_registration/run1.json" + ], + "notes": "no registration mechanism for this entry" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S3", + "arm": "S3/gitignore", + "root": ".agents/skills", + "exclusion": "gitignore", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/codex/daemon/S3/S3_gitignore/run1.json", + "out/codex/daemon/S3/S3_gitignore/run2.json" + ], + "notes": "hook_trust=none turn=completed tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S3", + "arm": "S3/info-exclude", + "root": ".agents/skills", + "exclusion": "info-exclude", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/codex/daemon/S3/S3_info-exclude/run1.json", + "out/codex/daemon/S3/S3_info-exclude/run2.json" + ], + "notes": "hook_trust=none turn=completed tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".agent/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/codex/daemon/S4/S4_all-roots/run10.json", + "out/codex/daemon/S4/S4_all-roots/run20.json" + ], + "notes": "found=['agents', 'codex'] leaked=[] hook_trust=none turn=completed tools_used=0 skill_reads=2 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/codex/daemon/S4/S4_all-roots/run12.json", + "out/codex/daemon/S4/S4_all-roots/run2.json" + ], + "notes": "found=['agents', 'codex'] leaked=[] hook_trust=none turn=completed tools_used=0 skill_reads=2 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".claude/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/codex/daemon/S4/S4_all-roots/run1.json", + "out/codex/daemon/S4/S4_all-roots/run11.json" + ], + "notes": "found=['agents', 'codex'] leaked=[] hook_trust=none turn=completed tools_used=0 skill_reads=2 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".codex/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/codex/daemon/S4/S4_all-roots/run13.json", + "out/codex/daemon/S4/S4_all-roots/run3.json" + ], + "notes": "found=['agents', 'codex'] leaked=[] hook_trust=none turn=completed tools_used=0 skill_reads=2 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/codex/daemon/S4/S4_all-roots/run14.json", + "out/codex/daemon/S4/S4_all-roots/run4.json" + ], + "notes": "found=['agents', 'codex'] leaked=[] hook_trust=none turn=completed tools_used=0 skill_reads=2 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/codex/daemon/S4/S4_all-roots/run16.json", + "out/codex/daemon/S4/S4_all-roots/run6.json" + ], + "notes": "found=['agents', 'codex'] leaked=[] hook_trust=none turn=completed tools_used=0 skill_reads=2 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".github/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/codex/daemon/S4/S4_all-roots/run15.json", + "out/codex/daemon/S4/S4_all-roots/run5.json" + ], + "notes": "found=['agents', 'codex'] leaked=[] hook_trust=none turn=completed tools_used=0 skill_reads=2 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/codex/daemon/S4/S4_all-roots/run17.json", + "out/codex/daemon/S4/S4_all-roots/run7.json" + ], + "notes": "found=['agents', 'codex'] leaked=[] hook_trust=none turn=completed tools_used=0 skill_reads=2 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/codex/daemon/S4/S4_all-roots/run19.json", + "out/codex/daemon/S4/S4_all-roots/run9.json" + ], + "notes": "found=['agents', 'codex'] leaked=[] hook_trust=none turn=completed tools_used=0 skill_reads=2 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/codex/daemon/S4/S4_all-roots/run18.json", + "out/codex/daemon/S4/S4_all-roots/run8.json" + ], + "notes": "found=['agents', 'codex'] leaked=[] hook_trust=none turn=completed tools_used=0 skill_reads=2 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-agent", + "root": ".agent/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/codex/daemon/S4/S4_confirm-agent/run1.json" + ], + "notes": "hook_trust=none turn=completed tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-claude", + "root": ".claude/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/codex/daemon/S4/S4_confirm-claude/run1.json" + ], + "notes": "hook_trust=none turn=completed tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-cursor", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/codex/daemon/S4/S4_confirm-cursor/run1.json" + ], + "notes": "hook_trust=none turn=completed tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-gemini", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/codex/daemon/S4/S4_confirm-gemini/run1.json" + ], + "notes": "hook_trust=none turn=completed tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-github", + "root": ".github/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/codex/daemon/S4/S4_confirm-github/run1.json" + ], + "notes": "hook_trust=none turn=completed tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-kiro", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/codex/daemon/S4/S4_confirm-kiro/run1.json" + ], + "notes": "hook_trust=none turn=completed tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-opencode", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/codex/daemon/S4/S4_confirm-opencode/run1.json" + ], + "notes": "hook_trust=none turn=completed tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-pi", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/codex/daemon/S4/S4_confirm-pi/run1.json" + ], + "notes": "hook_trust=none turn=completed tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S5", + "arm": "S5/add", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "visible_live", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/codex/daemon/S5/S5_add/run1.json", + "out/codex/daemon/S5/S5_add/run2.json" + ], + "notes": "turn1=not_visible hook_trust=none turn=completed tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S5", + "arm": "S5/delete", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "revoked", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/codex/daemon/S5/S5_delete/run1.json", + "out/codex/daemon/S5/S5_delete/run2.json" + ], + "notes": "turn1=visible_first_turn hook_trust=none turn=completed tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S5", + "arm": "S5/update", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "visible_live", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/codex/daemon/S5/S5_update/run1.json", + "out/codex/daemon/S5/S5_update/run2.json" + ], + "notes": "turn1=visible_first_turn hook_trust=none turn=completed tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S6", + "arm": "S6/delete", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "stale", + "flaky": false, + "runs": 2, + "mechanism": "hooks.json SessionStart", + "evidence": [ + "out/codex/daemon/S6/S6_delete/run1.json", + "out/codex/daemon/S6/S6_delete/run2.json" + ], + "notes": "hook_trust=seeded turn=completed tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S6", + "arm": "S6/update", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": "hooks.json SessionStart", + "evidence": [ + "out/codex/daemon/S6/S6_update/run1.json", + "out/codex/daemon/S6/S6_update/run2.json" + ], + "notes": "hook_trust=seeded turn=completed tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S0", + "arm": "S0/none", + "root": null, + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/codex/print/S0/S0_none/run1.json", + "out/codex/print/S0/S0_none/run2.json" + ], + "notes": "tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S1", + "arm": "S1/native", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/codex/print/S1/S1_native/run1.json", + "out/codex/print/S1/S1_native/run2.json" + ], + "notes": "tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S2", + "arm": "S2/hook-adds-skill", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": "hooks.json SessionStart", + "evidence": [ + "out/codex/print/S2/S2_hook-adds-skill/run1.json", + "out/codex/print/S2/S2_hook-adds-skill/run2.json" + ], + "notes": "tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S2", + "arm": "S2/hook-creates-root", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": "hooks.json SessionStart", + "evidence": [ + "out/codex/print/S2/S2_hook-creates-root/run1.json", + "out/codex/print/S2/S2_hook-creates-root/run2.json" + ], + "notes": "tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S2", + "arm": "S2/registration", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/codex/print/S2/S2_registration/run1.json" + ], + "notes": "no registration mechanism for this entry" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S3", + "arm": "S3/gitignore", + "root": ".agents/skills", + "exclusion": "gitignore", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/codex/print/S3/S3_gitignore/run1.json", + "out/codex/print/S3/S3_gitignore/run2.json" + ], + "notes": "tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S3", + "arm": "S3/info-exclude", + "root": ".agents/skills", + "exclusion": "info-exclude", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/codex/print/S3/S3_info-exclude/run1.json", + "out/codex/print/S3/S3_info-exclude/run2.json" + ], + "notes": "tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".agent/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/codex/print/S4/S4_all-roots/run10.json", + "out/codex/print/S4/S4_all-roots/run20.json" + ], + "notes": "found=['agents', 'codex'] leaked=[] tools_used=0 skill_reads=2 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/codex/print/S4/S4_all-roots/run12.json", + "out/codex/print/S4/S4_all-roots/run2.json" + ], + "notes": "found=['agents', 'codex'] leaked=[] tools_used=0 skill_reads=2 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".claude/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/codex/print/S4/S4_all-roots/run1.json", + "out/codex/print/S4/S4_all-roots/run11.json" + ], + "notes": "found=['agents', 'codex'] leaked=[] tools_used=0 skill_reads=2 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".codex/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/codex/print/S4/S4_all-roots/run13.json", + "out/codex/print/S4/S4_all-roots/run3.json" + ], + "notes": "found=['agents', 'codex'] leaked=[] tools_used=0 skill_reads=2 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/codex/print/S4/S4_all-roots/run14.json", + "out/codex/print/S4/S4_all-roots/run4.json" + ], + "notes": "found=['agents', 'codex'] leaked=[] tools_used=0 skill_reads=2 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/codex/print/S4/S4_all-roots/run16.json", + "out/codex/print/S4/S4_all-roots/run6.json" + ], + "notes": "found=['agents', 'codex'] leaked=[] tools_used=0 skill_reads=2 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".github/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/codex/print/S4/S4_all-roots/run15.json", + "out/codex/print/S4/S4_all-roots/run5.json" + ], + "notes": "found=['agents', 'codex'] leaked=[] tools_used=0 skill_reads=2 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/codex/print/S4/S4_all-roots/run17.json", + "out/codex/print/S4/S4_all-roots/run7.json" + ], + "notes": "found=['agents', 'codex'] leaked=[] tools_used=0 skill_reads=2 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/codex/print/S4/S4_all-roots/run19.json", + "out/codex/print/S4/S4_all-roots/run9.json" + ], + "notes": "found=['agents', 'codex'] leaked=[] tools_used=0 skill_reads=2 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/codex/print/S4/S4_all-roots/run18.json", + "out/codex/print/S4/S4_all-roots/run8.json" + ], + "notes": "found=['agents', 'codex'] leaked=[] tools_used=0 skill_reads=2 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-agent", + "root": ".agent/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/codex/print/S4/S4_confirm-agent/run1.json" + ], + "notes": "tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-claude", + "root": ".claude/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/codex/print/S4/S4_confirm-claude/run1.json" + ], + "notes": "tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-cursor", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/codex/print/S4/S4_confirm-cursor/run1.json" + ], + "notes": "tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-gemini", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/codex/print/S4/S4_confirm-gemini/run1.json" + ], + "notes": "tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-github", + "root": ".github/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/codex/print/S4/S4_confirm-github/run1.json" + ], + "notes": "tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-kiro", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/codex/print/S4/S4_confirm-kiro/run1.json" + ], + "notes": "tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-opencode", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/codex/print/S4/S4_confirm-opencode/run1.json" + ], + "notes": "tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-pi", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/codex/print/S4/S4_confirm-pi/run1.json" + ], + "notes": "tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S6", + "arm": "S6/delete", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": "hooks.json SessionStart", + "evidence": [ + "out/codex/print/S6/S6_delete/run1.json", + "out/codex/print/S6/S6_delete/run2.json" + ], + "notes": "tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S6", + "arm": "S6/update", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": "hooks.json SessionStart", + "evidence": [ + "out/codex/print/S6/S6_update/run1.json", + "out/codex/print/S6/S6_update/run2.json" + ], + "notes": "tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S7", + "arm": "S7/add", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/codex/print/S7/S7_add/run1.json", + "out/codex/print/S7/S7_add/run2.json" + ], + "notes": "turn1=not_visible session=01a0af6d-4939-7a02-b28b-319c822092ee tools_used=0 skill_reads=1 searches=0; turn1=not_visible session=01a0af6d-9188-7cf2-bfb5-2508a6351db4 tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S7", + "arm": "S7/update", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/codex/print/S7/S7_update/run1.json", + "out/codex/print/S7/S7_update/run2.json" + ], + "notes": "turn1=visible_first_turn session=01a0af6d-cd91-7961-b88d-14c7db4cdf96 tools_used=0 skill_reads=1 searches=0; turn1=visible_first_turn session=01a0af6e-27da-7620-b906-9e05eb31c3ab tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S8", + "arm": "S8/ancestor", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/codex/print/S8/S8_ancestor/run1.json", + "out/codex/print/S8/S8_ancestor/run2.json" + ], + "notes": "cwd=sub/dir tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S8", + "arm": "S8/local", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/codex/print/S8/S8_local/run1.json", + "out/codex/print/S8/S8_local/run2.json" + ], + "notes": "cwd=sub/dir tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S9", + "arm": "S9/linked-other", + "root": ".agents/skills", + "exclusion": "info-exclude", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/codex/print/S9/S9_linked-other/run1.json", + "out/codex/print/S9/S9_linked-other/run2.json" + ], + "notes": "cwd=wt-b skill_in=main tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S9", + "arm": "S9/linked-own", + "root": ".agents/skills", + "exclusion": "info-exclude", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/codex/print/S9/S9_linked-own/run1.json", + "out/codex/print/S9/S9_linked-own/run2.json" + ], + "notes": "cwd=wt-b skill_in=linked tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S0", + "arm": "S0/none", + "root": null, + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/codex/tui/S0/S0_none/run1.json", + "out/codex/tui/S0/S0_none/run2.json" + ], + "notes": "" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S1", + "arm": "S1/native", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/codex/tui/S1/S1_native/run1.json", + "out/codex/tui/S1/S1_native/run2.json" + ], + "notes": "" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S2", + "arm": "S2/hook-adds-skill", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": "hooks.json SessionStart", + "evidence": [ + "out/codex/tui/S2/S2_hook-adds-skill/run1.json", + "out/codex/tui/S2/S2_hook-adds-skill/run2.json" + ], + "notes": "" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S5", + "arm": "S5/add", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "visible_live", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/codex/tui/S5/S5_add/run1.json", + "out/codex/tui/S5/S5_add/run2.json" + ], + "notes": "turn1=not_visible" + }, + { + "entry": "codex", + "harness": "codex", + "version": "codex-cli 0.154.0", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S5", + "arm": "S5/reload", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/codex/tui/S5/S5_reload/run1.json" + ], + "notes": "no reload command for this entry" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S0", + "arm": "S0/none", + "root": null, + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/copilot/daemon/S0/S0_none/run1.json", + "out/copilot/daemon/S0/S0_none/run2.json" + ], + "notes": "stopReason=end_turn tools_used=0; stopReason=end_turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S1", + "arm": "S1/native", + "root": ".github/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/copilot/daemon/S1/S1_native/run1.json", + "out/copilot/daemon/S1/S1_native/run2.json" + ], + "notes": "stopReason=end_turn tools_used=1; stopReason=end_turn tools_used=0 skill_loads=1 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S10", + "arm": "S10/hook-from-peer", + "root": ".github/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": "hooks/*.json sessionStart", + "evidence": [ + "out/copilot/daemon/S10/S10_hook-from-peer/run1.json", + "out/copilot/daemon/S10/S10_hook-from-peer/run2.json" + ], + "notes": "turn1=not_visible peer=not_visible stopReason=end_turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S2", + "arm": "S2/hook-adds-skill", + "root": ".github/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": "hooks/*.json sessionStart", + "evidence": [ + "out/copilot/daemon/S2/S2_hook-adds-skill/run1.json", + "out/copilot/daemon/S2/S2_hook-adds-skill/run2.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S2", + "arm": "S2/hook-creates-root", + "root": ".github/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": "hooks/*.json sessionStart", + "evidence": [ + "out/copilot/daemon/S2/S2_hook-creates-root/run1.json", + "out/copilot/daemon/S2/S2_hook-creates-root/run2.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S2", + "arm": "S2/registration", + "root": ".github/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/copilot/daemon/S2/S2_registration/run1.json" + ], + "notes": "no registration mechanism for this entry" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S3", + "arm": "S3/gitignore", + "root": ".github/skills", + "exclusion": "gitignore", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/copilot/daemon/S3/S3_gitignore/run1.json", + "out/copilot/daemon/S3/S3_gitignore/run2.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_loads=1 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S3", + "arm": "S3/info-exclude", + "root": ".github/skills", + "exclusion": "info-exclude", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/copilot/daemon/S3/S3_info-exclude/run1.json", + "out/copilot/daemon/S3/S3_info-exclude/run2.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_loads=1 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".agent/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/copilot/daemon/S4/S4_all-roots/run10.json", + "out/copilot/daemon/S4/S4_all-roots/run20.json" + ], + "notes": "found=['agents', 'claude', 'github'] leaked=[] stopReason=end_turn tools_used=0 skill_loads=3 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/copilot/daemon/S4/S4_all-roots/run12.json", + "out/copilot/daemon/S4/S4_all-roots/run2.json" + ], + "notes": "found=['agents', 'claude', 'github'] leaked=[] stopReason=end_turn tools_used=0 skill_loads=3 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".claude/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/copilot/daemon/S4/S4_all-roots/run1.json", + "out/copilot/daemon/S4/S4_all-roots/run11.json" + ], + "notes": "found=['agents', 'claude', 'github'] leaked=[] stopReason=end_turn tools_used=0 skill_loads=3 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".codex/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/copilot/daemon/S4/S4_all-roots/run13.json", + "out/copilot/daemon/S4/S4_all-roots/run3.json" + ], + "notes": "found=['agents', 'claude', 'github'] leaked=[] stopReason=end_turn tools_used=0 skill_loads=3 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/copilot/daemon/S4/S4_all-roots/run14.json", + "out/copilot/daemon/S4/S4_all-roots/run4.json" + ], + "notes": "found=['agents', 'claude', 'github'] leaked=[] stopReason=end_turn tools_used=0 skill_loads=3 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/copilot/daemon/S4/S4_all-roots/run16.json", + "out/copilot/daemon/S4/S4_all-roots/run6.json" + ], + "notes": "found=['agents', 'claude', 'github'] leaked=[] stopReason=end_turn tools_used=0 skill_loads=3 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".github/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/copilot/daemon/S4/S4_all-roots/run15.json", + "out/copilot/daemon/S4/S4_all-roots/run5.json" + ], + "notes": "found=['agents', 'claude', 'github'] leaked=[] stopReason=end_turn tools_used=0 skill_loads=3 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/copilot/daemon/S4/S4_all-roots/run17.json", + "out/copilot/daemon/S4/S4_all-roots/run7.json" + ], + "notes": "found=['agents', 'claude', 'github'] leaked=[] stopReason=end_turn tools_used=0 skill_loads=3 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/copilot/daemon/S4/S4_all-roots/run19.json", + "out/copilot/daemon/S4/S4_all-roots/run9.json" + ], + "notes": "found=['agents', 'claude', 'github'] leaked=[] stopReason=end_turn tools_used=0 skill_loads=3 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/copilot/daemon/S4/S4_all-roots/run18.json", + "out/copilot/daemon/S4/S4_all-roots/run8.json" + ], + "notes": "found=['agents', 'claude', 'github'] leaked=[] stopReason=end_turn tools_used=0 skill_loads=3 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-agent", + "root": ".agent/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/copilot/daemon/S4/S4_confirm-agent/run1.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-codex", + "root": ".codex/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/copilot/daemon/S4/S4_confirm-codex/run1.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-cursor", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/copilot/daemon/S4/S4_confirm-cursor/run1.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-gemini", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/copilot/daemon/S4/S4_confirm-gemini/run1.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-kiro", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/copilot/daemon/S4/S4_confirm-kiro/run1.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-opencode", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/copilot/daemon/S4/S4_confirm-opencode/run1.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-pi", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/copilot/daemon/S4/S4_confirm-pi/run1.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S5", + "arm": "S5/add", + "root": ".github/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/copilot/daemon/S5/S5_add/run1.json", + "out/copilot/daemon/S5/S5_add/run2.json" + ], + "notes": "turn1=not_visible stopReason=end_turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S5", + "arm": "S5/delete", + "root": ".github/skills", + "exclusion": "none", + "verdict": "stale", + "flaky": true, + "runs": 3, + "mechanism": null, + "evidence": [ + "out/copilot/daemon/S5/S5_delete/run1.json", + "out/copilot/daemon/S5/S5_delete/run2.json", + "out/copilot/daemon/S5/S5_delete/run3.json" + ], + "notes": "turn1=visible_first_turn stopReason=end_turn tools_used=0 skill_loads=0 searches=0; turn1=visible_first_turn stopReason=end_turn tools_used=0 skill_loads=1 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S5", + "arm": "S5/update", + "root": ".github/skills", + "exclusion": "none", + "verdict": "stale", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/copilot/daemon/S5/S5_update/run1.json", + "out/copilot/daemon/S5/S5_update/run2.json" + ], + "notes": "turn1=visible_first_turn stopReason=end_turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S6", + "arm": "S6/delete", + "root": ".github/skills", + "exclusion": "none", + "verdict": "revoked", + "flaky": false, + "runs": 2, + "mechanism": "hooks/*.json sessionStart", + "evidence": [ + "out/copilot/daemon/S6/S6_delete/run1.json", + "out/copilot/daemon/S6/S6_delete/run2.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_loads=1 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S6", + "arm": "S6/update", + "root": ".github/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": "hooks/*.json sessionStart", + "evidence": [ + "out/copilot/daemon/S6/S6_update/run1.json", + "out/copilot/daemon/S6/S6_update/run2.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_loads=1 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S0", + "arm": "S0/none", + "root": null, + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/copilot/print/S0/S0_none/run1.json", + "out/copilot/print/S0/S0_none/run2.json" + ], + "notes": "tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S1", + "arm": "S1/native", + "root": ".github/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/copilot/print/S1/S1_native/run1.json", + "out/copilot/print/S1/S1_native/run2.json" + ], + "notes": "tools_used=0 skill_loads=1 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S2", + "arm": "S2/hook-adds-skill", + "root": ".github/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": "hooks/*.json sessionStart", + "evidence": [ + "out/copilot/print/S2/S2_hook-adds-skill/run1.json", + "out/copilot/print/S2/S2_hook-adds-skill/run2.json" + ], + "notes": "tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S2", + "arm": "S2/hook-creates-root", + "root": ".github/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": "hooks/*.json sessionStart", + "evidence": [ + "out/copilot/print/S2/S2_hook-creates-root/run1.json", + "out/copilot/print/S2/S2_hook-creates-root/run2.json" + ], + "notes": "tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S2", + "arm": "S2/registration", + "root": ".github/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/copilot/print/S2/S2_registration/run1.json" + ], + "notes": "no registration mechanism for this entry" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S3", + "arm": "S3/gitignore", + "root": ".github/skills", + "exclusion": "gitignore", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/copilot/print/S3/S3_gitignore/run1.json", + "out/copilot/print/S3/S3_gitignore/run2.json" + ], + "notes": "tools_used=0 skill_loads=1 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S3", + "arm": "S3/info-exclude", + "root": ".github/skills", + "exclusion": "info-exclude", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/copilot/print/S3/S3_info-exclude/run1.json", + "out/copilot/print/S3/S3_info-exclude/run2.json" + ], + "notes": "tools_used=0 skill_loads=1 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".agent/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/copilot/print/S4/S4_all-roots/run10.json", + "out/copilot/print/S4/S4_all-roots/run20.json" + ], + "notes": "found=['agents', 'claude', 'github'] leaked=[] tools_used=0 skill_loads=3 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/copilot/print/S4/S4_all-roots/run12.json", + "out/copilot/print/S4/S4_all-roots/run2.json" + ], + "notes": "found=['agents', 'claude', 'github'] leaked=[] tools_used=0 skill_loads=3 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".claude/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/copilot/print/S4/S4_all-roots/run1.json", + "out/copilot/print/S4/S4_all-roots/run11.json" + ], + "notes": "found=['agents', 'claude', 'github'] leaked=[] tools_used=0 skill_loads=3 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".codex/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/copilot/print/S4/S4_all-roots/run13.json", + "out/copilot/print/S4/S4_all-roots/run3.json" + ], + "notes": "found=['agents', 'claude', 'github'] leaked=[] tools_used=0 skill_loads=3 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/copilot/print/S4/S4_all-roots/run14.json", + "out/copilot/print/S4/S4_all-roots/run4.json" + ], + "notes": "found=['agents', 'claude', 'github'] leaked=[] tools_used=0 skill_loads=3 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/copilot/print/S4/S4_all-roots/run16.json", + "out/copilot/print/S4/S4_all-roots/run6.json" + ], + "notes": "found=['agents', 'claude', 'github'] leaked=[] tools_used=0 skill_loads=3 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".github/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/copilot/print/S4/S4_all-roots/run15.json", + "out/copilot/print/S4/S4_all-roots/run5.json" + ], + "notes": "found=['agents', 'claude', 'github'] leaked=[] tools_used=0 skill_loads=3 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/copilot/print/S4/S4_all-roots/run17.json", + "out/copilot/print/S4/S4_all-roots/run7.json" + ], + "notes": "found=['agents', 'claude', 'github'] leaked=[] tools_used=0 skill_loads=3 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/copilot/print/S4/S4_all-roots/run19.json", + "out/copilot/print/S4/S4_all-roots/run9.json" + ], + "notes": "found=['agents', 'claude', 'github'] leaked=[] tools_used=0 skill_loads=3 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/copilot/print/S4/S4_all-roots/run18.json", + "out/copilot/print/S4/S4_all-roots/run8.json" + ], + "notes": "found=['agents', 'claude', 'github'] leaked=[] tools_used=0 skill_loads=3 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-agent", + "root": ".agent/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/copilot/print/S4/S4_confirm-agent/run1.json" + ], + "notes": "tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-codex", + "root": ".codex/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/copilot/print/S4/S4_confirm-codex/run1.json" + ], + "notes": "tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-cursor", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/copilot/print/S4/S4_confirm-cursor/run1.json" + ], + "notes": "tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-gemini", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/copilot/print/S4/S4_confirm-gemini/run1.json" + ], + "notes": "tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-kiro", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/copilot/print/S4/S4_confirm-kiro/run1.json" + ], + "notes": "tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-opencode", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/copilot/print/S4/S4_confirm-opencode/run1.json" + ], + "notes": "tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-pi", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/copilot/print/S4/S4_confirm-pi/run1.json" + ], + "notes": "tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S6", + "arm": "S6/delete", + "root": ".github/skills", + "exclusion": "none", + "verdict": "revoked", + "flaky": false, + "runs": 2, + "mechanism": "hooks/*.json sessionStart", + "evidence": [ + "out/copilot/print/S6/S6_delete/run1.json", + "out/copilot/print/S6/S6_delete/run2.json" + ], + "notes": "tools_used=0 skill_loads=1 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S6", + "arm": "S6/update", + "root": ".github/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": "hooks/*.json sessionStart", + "evidence": [ + "out/copilot/print/S6/S6_update/run1.json", + "out/copilot/print/S6/S6_update/run2.json" + ], + "notes": "tools_used=0 skill_loads=1 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S7", + "arm": "S7/add", + "root": ".github/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/copilot/print/S7/S7_add/run1.json", + "out/copilot/print/S7/S7_add/run2.json" + ], + "notes": "turn1=not_visible session=180a58ea-c5cd-485f-8839-19b99e6f675a tools_used=0 skill_loads=1 searches=0; turn1=not_visible session=06cb4588-428d-4ae2-872f-5f0237e01319 tools_used=0 skill_loads=1 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S7", + "arm": "S7/update", + "root": ".github/skills", + "exclusion": "none", + "verdict": "stale", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/copilot/print/S7/S7_update/run1.json", + "out/copilot/print/S7/S7_update/run2.json" + ], + "notes": "turn1=visible_first_turn session=f34ad7cc-a08c-4267-b8c4-de2a0f3ba691 tools_used=0 skill_loads=0 searches=0; turn1=visible_first_turn session=b7b5c7a7-2ae6-4612-8e66-a61c8c7854a2 tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S8", + "arm": "S8/ancestor", + "root": ".github/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/copilot/print/S8/S8_ancestor/run1.json", + "out/copilot/print/S8/S8_ancestor/run2.json" + ], + "notes": "cwd=sub/dir tools_used=0 skill_loads=1 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S8", + "arm": "S8/local", + "root": ".github/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/copilot/print/S8/S8_local/run1.json", + "out/copilot/print/S8/S8_local/run2.json" + ], + "notes": "cwd=sub/dir tools_used=0 skill_loads=1 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S9", + "arm": "S9/linked-other", + "root": ".github/skills", + "exclusion": "info-exclude", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/copilot/print/S9/S9_linked-other/run1.json", + "out/copilot/print/S9/S9_linked-other/run2.json" + ], + "notes": "cwd=wt-b skill_in=main tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S9", + "arm": "S9/linked-own", + "root": ".github/skills", + "exclusion": "info-exclude", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/copilot/print/S9/S9_linked-own/run1.json", + "out/copilot/print/S9/S9_linked-own/run2.json" + ], + "notes": "cwd=wt-b skill_in=linked tools_used=0 skill_loads=1 searches=0" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S0", + "arm": "S0/none", + "root": null, + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/copilot/tui/S0/S0_none/run1.json", + "out/copilot/tui/S0/S0_none/run2.json" + ], + "notes": "dialog=Do you trust the files" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S1", + "arm": "S1/native", + "root": ".github/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/copilot/tui/S1/S1_native/run1.json", + "out/copilot/tui/S1/S1_native/run2.json" + ], + "notes": "dialog=Do you trust the files" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S2", + "arm": "S2/hook-adds-skill", + "root": ".github/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": "hooks/*.json sessionStart", + "evidence": [ + "out/copilot/tui/S2/S2_hook-adds-skill/run1.json", + "out/copilot/tui/S2/S2_hook-adds-skill/run2.json" + ], + "notes": "dialog=Do you trust the files" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S5", + "arm": "S5/add", + "root": ".github/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/copilot/tui/S5/S5_add/run1.json", + "out/copilot/tui/S5/S5_add/run2.json" + ], + "notes": "turn1=not_visible" + }, + { + "entry": "copilot", + "harness": "copilot", + "version": "GitHub Copilot CLI 1.0.85.", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S5", + "arm": "S5/reload", + "root": ".github/skills", + "exclusion": "none", + "verdict": "visible_after_reload", + "flaky": false, + "runs": 2, + "mechanism": "/skills reload", + "evidence": [ + "out/copilot/tui/S5/S5_reload/run1.json", + "out/copilot/tui/S5/S5_reload/run2.json" + ], + "notes": "turn1=not_visible" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S0", + "arm": "S0/none", + "root": null, + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/cursor/daemon/S0/S0_none/run1.json", + "out/cursor/daemon/S0/S0_none/run2.json" + ], + "notes": "stopReason=end_turn tools_used=0" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S1", + "arm": "S1/native", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/cursor/daemon/S1/S1_native/run1.json", + "out/cursor/daemon/S1/S1_native/run2.json" + ], + "notes": "stopReason=end_turn tools_used=1" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S10", + "arm": "S10/hook-from-peer", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": "project .cursor/hooks.json sessionStart", + "evidence": [ + "out/cursor/daemon/S10/S10_hook-from-peer/run1.json", + "out/cursor/daemon/S10/S10_hook-from-peer/run2.json" + ], + "notes": "turn1=not_visible peer=not_visible stopReason=end_turn tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S2", + "arm": "S2/hook-adds-skill", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 2, + "mechanism": "project .cursor/hooks.json sessionStart", + "evidence": [ + "out/cursor/daemon/S2/S2_hook-adds-skill/run1.json", + "out/cursor/daemon/S2/S2_hook-adds-skill/run2.json" + ], + "notes": "hook never fired skill file absent after the turn stopReason=end_turn tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S2", + "arm": "S2/hook-creates-root", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 2, + "mechanism": "project .cursor/hooks.json sessionStart", + "evidence": [ + "out/cursor/daemon/S2/S2_hook-creates-root/run1.json", + "out/cursor/daemon/S2/S2_hook-creates-root/run2.json" + ], + "notes": "hook never fired skill file absent after the turn stopReason=end_turn tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S2", + "arm": "S2/registration", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 1, + "mechanism": "project .cursor/hooks.json workspaceOpen pluginPaths", + "evidence": [ + "out/cursor/daemon/S2/S2_registration/run1.json" + ], + "notes": "skill file absent after the turn stopReason=end_turn tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S3", + "arm": "S3/gitignore", + "root": ".cursor/skills", + "exclusion": "gitignore", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/cursor/daemon/S3/S3_gitignore/run1.json", + "out/cursor/daemon/S3/S3_gitignore/run2.json" + ], + "notes": "stopReason=end_turn tools_used=1" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S3", + "arm": "S3/info-exclude", + "root": ".cursor/skills", + "exclusion": "info-exclude", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/cursor/daemon/S3/S3_info-exclude/run1.json", + "out/cursor/daemon/S3/S3_info-exclude/run2.json" + ], + "notes": "stopReason=end_turn tools_used=1" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".agent/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/cursor/daemon/S4/S4_all-roots/run10.json", + "out/cursor/daemon/S4/S4_all-roots/run20.json" + ], + "notes": "found=['agents', 'claude', 'codex', 'cursor'] leaked=[] stopReason=end_turn tools_used=4" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/cursor/daemon/S4/S4_all-roots/run12.json", + "out/cursor/daemon/S4/S4_all-roots/run2.json" + ], + "notes": "found=['agents', 'claude', 'codex', 'cursor'] leaked=[] stopReason=end_turn tools_used=4" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".claude/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/cursor/daemon/S4/S4_all-roots/run1.json", + "out/cursor/daemon/S4/S4_all-roots/run11.json" + ], + "notes": "found=['agents', 'claude', 'codex', 'cursor'] leaked=[] stopReason=end_turn tools_used=4" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".codex/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/cursor/daemon/S4/S4_all-roots/run13.json", + "out/cursor/daemon/S4/S4_all-roots/run3.json" + ], + "notes": "found=['agents', 'claude', 'codex', 'cursor'] leaked=[] stopReason=end_turn tools_used=4" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/cursor/daemon/S4/S4_all-roots/run14.json", + "out/cursor/daemon/S4/S4_all-roots/run4.json" + ], + "notes": "found=['agents', 'claude', 'codex', 'cursor'] leaked=[] stopReason=end_turn tools_used=4" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/cursor/daemon/S4/S4_all-roots/run16.json", + "out/cursor/daemon/S4/S4_all-roots/run6.json" + ], + "notes": "found=['agents', 'claude', 'codex', 'cursor'] leaked=[] stopReason=end_turn tools_used=4" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".github/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/cursor/daemon/S4/S4_all-roots/run15.json", + "out/cursor/daemon/S4/S4_all-roots/run5.json" + ], + "notes": "found=['agents', 'claude', 'codex', 'cursor'] leaked=[] stopReason=end_turn tools_used=4" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/cursor/daemon/S4/S4_all-roots/run17.json", + "out/cursor/daemon/S4/S4_all-roots/run7.json" + ], + "notes": "found=['agents', 'claude', 'codex', 'cursor'] leaked=[] stopReason=end_turn tools_used=4" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/cursor/daemon/S4/S4_all-roots/run19.json", + "out/cursor/daemon/S4/S4_all-roots/run9.json" + ], + "notes": "found=['agents', 'claude', 'codex', 'cursor'] leaked=[] stopReason=end_turn tools_used=4" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/cursor/daemon/S4/S4_all-roots/run18.json", + "out/cursor/daemon/S4/S4_all-roots/run8.json" + ], + "notes": "found=['agents', 'claude', 'codex', 'cursor'] leaked=[] stopReason=end_turn tools_used=4" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-agent", + "root": ".agent/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/cursor/daemon/S4/S4_confirm-agent/run1.json" + ], + "notes": "stopReason=end_turn tools_used=0" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-gemini", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/cursor/daemon/S4/S4_confirm-gemini/run1.json" + ], + "notes": "stopReason=end_turn tools_used=0" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-github", + "root": ".github/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/cursor/daemon/S4/S4_confirm-github/run1.json" + ], + "notes": "stopReason=end_turn tools_used=0" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-kiro", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/cursor/daemon/S4/S4_confirm-kiro/run1.json" + ], + "notes": "stopReason=end_turn tools_used=0" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-opencode", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/cursor/daemon/S4/S4_confirm-opencode/run1.json" + ], + "notes": "stopReason=end_turn tools_used=0" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-pi", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/cursor/daemon/S4/S4_confirm-pi/run1.json" + ], + "notes": "stopReason=end_turn tools_used=0" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S5", + "arm": "S5/add", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/cursor/daemon/S5/S5_add/run1.json", + "out/cursor/daemon/S5/S5_add/run2.json" + ], + "notes": "turn1=not_visible stopReason=end_turn tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S5", + "arm": "S5/delete", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "revoked", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/cursor/daemon/S5/S5_delete/run1.json", + "out/cursor/daemon/S5/S5_delete/run2.json" + ], + "notes": "turn1=visible_first_turn stopReason=end_turn tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S5", + "arm": "S5/update", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "visible_live", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/cursor/daemon/S5/S5_update/run1.json", + "out/cursor/daemon/S5/S5_update/run2.json" + ], + "notes": "turn1=visible_first_turn stopReason=end_turn tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S6", + "arm": "S6/delete", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 2, + "mechanism": "project .cursor/hooks.json sessionStart", + "evidence": [ + "out/cursor/daemon/S6/S6_delete/run1.json", + "out/cursor/daemon/S6/S6_delete/run2.json" + ], + "notes": "hook never fired stopReason=end_turn tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S6", + "arm": "S6/update", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 2, + "mechanism": "project .cursor/hooks.json sessionStart", + "evidence": [ + "out/cursor/daemon/S6/S6_update/run1.json", + "out/cursor/daemon/S6/S6_update/run2.json" + ], + "notes": "hook never fired stopReason=end_turn tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S0", + "arm": "S0/none", + "root": null, + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/cursor/print/S0/S0_none/run1.json", + "out/cursor/print/S0/S0_none/run2.json" + ], + "notes": "tools=unobserved" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S1", + "arm": "S1/native", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/cursor/print/S1/S1_native/run1.json", + "out/cursor/print/S1/S1_native/run2.json" + ], + "notes": "tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S2", + "arm": "S2/hook-adds-skill", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": "project .cursor/hooks.json sessionStart", + "evidence": [ + "out/cursor/print/S2/S2_hook-adds-skill/run1.json", + "out/cursor/print/S2/S2_hook-adds-skill/run2.json" + ], + "notes": "tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S2", + "arm": "S2/hook-creates-root", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": "project .cursor/hooks.json sessionStart", + "evidence": [ + "out/cursor/print/S2/S2_hook-creates-root/run1.json", + "out/cursor/print/S2/S2_hook-creates-root/run2.json" + ], + "notes": "tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S2", + "arm": "S2/registration", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "visible_after_reload", + "flaky": false, + "runs": 2, + "mechanism": "project .cursor/hooks.json workspaceOpen pluginPaths", + "evidence": [ + "out/cursor/print/S2/S2_registration/run1.json", + "out/cursor/print/S2/S2_registration/run2.json" + ], + "notes": "tools_used=1 skill_reads=1 searches=1; tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S3", + "arm": "S3/gitignore", + "root": ".cursor/skills", + "exclusion": "gitignore", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/cursor/print/S3/S3_gitignore/run1.json", + "out/cursor/print/S3/S3_gitignore/run2.json" + ], + "notes": "tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S3", + "arm": "S3/info-exclude", + "root": ".cursor/skills", + "exclusion": "info-exclude", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/cursor/print/S3/S3_info-exclude/run1.json", + "out/cursor/print/S3/S3_info-exclude/run2.json" + ], + "notes": "tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".agent/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/cursor/print/S4/S4_all-roots/run10.json", + "out/cursor/print/S4/S4_all-roots/run20.json" + ], + "notes": "found=['agents', 'claude', 'codex', 'cursor'] leaked=[] tools_used=0 skill_reads=4 searches=0" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/cursor/print/S4/S4_all-roots/run12.json", + "out/cursor/print/S4/S4_all-roots/run2.json" + ], + "notes": "found=['agents', 'claude', 'codex', 'cursor'] leaked=[] tools_used=0 skill_reads=4 searches=0" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".claude/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/cursor/print/S4/S4_all-roots/run1.json", + "out/cursor/print/S4/S4_all-roots/run11.json" + ], + "notes": "found=['agents', 'claude', 'codex', 'cursor'] leaked=[] tools_used=0 skill_reads=4 searches=0" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".codex/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/cursor/print/S4/S4_all-roots/run13.json", + "out/cursor/print/S4/S4_all-roots/run3.json" + ], + "notes": "found=['agents', 'claude', 'codex', 'cursor'] leaked=[] tools_used=0 skill_reads=4 searches=0" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/cursor/print/S4/S4_all-roots/run14.json", + "out/cursor/print/S4/S4_all-roots/run4.json" + ], + "notes": "found=['agents', 'claude', 'codex', 'cursor'] leaked=[] tools_used=0 skill_reads=4 searches=0" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/cursor/print/S4/S4_all-roots/run16.json", + "out/cursor/print/S4/S4_all-roots/run6.json" + ], + "notes": "found=['agents', 'claude', 'codex', 'cursor'] leaked=[] tools_used=0 skill_reads=4 searches=0" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".github/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/cursor/print/S4/S4_all-roots/run15.json", + "out/cursor/print/S4/S4_all-roots/run5.json" + ], + "notes": "found=['agents', 'claude', 'codex', 'cursor'] leaked=[] tools_used=0 skill_reads=4 searches=0" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/cursor/print/S4/S4_all-roots/run17.json", + "out/cursor/print/S4/S4_all-roots/run7.json" + ], + "notes": "found=['agents', 'claude', 'codex', 'cursor'] leaked=[] tools_used=0 skill_reads=4 searches=0" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/cursor/print/S4/S4_all-roots/run19.json", + "out/cursor/print/S4/S4_all-roots/run9.json" + ], + "notes": "found=['agents', 'claude', 'codex', 'cursor'] leaked=[] tools_used=0 skill_reads=4 searches=0" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/cursor/print/S4/S4_all-roots/run18.json", + "out/cursor/print/S4/S4_all-roots/run8.json" + ], + "notes": "found=['agents', 'claude', 'codex', 'cursor'] leaked=[] tools_used=0 skill_reads=4 searches=0" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-agent", + "root": ".agent/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/cursor/print/S4/S4_confirm-agent/run1.json" + ], + "notes": "tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-gemini", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/cursor/print/S4/S4_confirm-gemini/run1.json" + ], + "notes": "tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-github", + "root": ".github/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/cursor/print/S4/S4_confirm-github/run1.json" + ], + "notes": "tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-kiro", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/cursor/print/S4/S4_confirm-kiro/run1.json" + ], + "notes": "tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-opencode", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/cursor/print/S4/S4_confirm-opencode/run1.json" + ], + "notes": "tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-pi", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/cursor/print/S4/S4_confirm-pi/run1.json" + ], + "notes": "tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S6", + "arm": "S6/delete", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "revoked", + "flaky": false, + "runs": 2, + "mechanism": "project .cursor/hooks.json sessionStart", + "evidence": [ + "out/cursor/print/S6/S6_delete/run1.json", + "out/cursor/print/S6/S6_delete/run2.json" + ], + "notes": "tools_used=3 skill_reads=3 searches=3; tools_used=1 skill_reads=2 searches=1" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S6", + "arm": "S6/update", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": "project .cursor/hooks.json sessionStart", + "evidence": [ + "out/cursor/print/S6/S6_update/run1.json", + "out/cursor/print/S6/S6_update/run2.json" + ], + "notes": "tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S7", + "arm": "S7/add", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/cursor/print/S7/S7_add/run1.json", + "out/cursor/print/S7/S7_add/run2.json" + ], + "notes": "turn1=not_visible session=4f5d5553-3029-46fb-b1ac-3aa800d07dbb tools_used=0 skill_reads=0 searches=0; turn1=not_visible session=3fc8e508-0d85-4726-9882-6ea80300ec06 tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S7", + "arm": "S7/update", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/cursor/print/S7/S7_update/run1.json", + "out/cursor/print/S7/S7_update/run2.json" + ], + "notes": "turn1=visible_first_turn session=2eef1089-a2b7-48f1-b84f-88bc27ae98cd tools_used=0 skill_reads=1 searches=0; turn1=visible_first_turn session=4725e951-3d8b-4873-8d24-3036c75f25c7 tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S8", + "arm": "S8/ancestor", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/cursor/print/S8/S8_ancestor/run1.json", + "out/cursor/print/S8/S8_ancestor/run2.json" + ], + "notes": "cwd=sub/dir tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S8", + "arm": "S8/local", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/cursor/print/S8/S8_local/run1.json", + "out/cursor/print/S8/S8_local/run2.json" + ], + "notes": "cwd=sub/dir tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S9", + "arm": "S9/linked-other", + "root": ".cursor/skills", + "exclusion": "info-exclude", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/cursor/print/S9/S9_linked-other/run1.json", + "out/cursor/print/S9/S9_linked-other/run2.json" + ], + "notes": "cwd=wt-b skill_in=main tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S9", + "arm": "S9/linked-own", + "root": ".cursor/skills", + "exclusion": "info-exclude", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/cursor/print/S9/S9_linked-own/run1.json", + "out/cursor/print/S9/S9_linked-own/run2.json" + ], + "notes": "cwd=wt-b skill_in=linked tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S0", + "arm": "S0/none", + "root": null, + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/cursor/tui/S0/S0_none/run1.json", + "out/cursor/tui/S0/S0_none/run2.json" + ], + "notes": "" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S1", + "arm": "S1/native", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/cursor/tui/S1/S1_native/run1.json", + "out/cursor/tui/S1/S1_native/run2.json" + ], + "notes": "" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S2", + "arm": "S2/hook-adds-skill", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": "project .cursor/hooks.json sessionStart", + "evidence": [ + "out/cursor/tui/S2/S2_hook-adds-skill/run1.json", + "out/cursor/tui/S2/S2_hook-adds-skill/run2.json" + ], + "notes": "" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S5", + "arm": "S5/add", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/cursor/tui/S5/S5_add/run1.json", + "out/cursor/tui/S5/S5_add/run2.json" + ], + "notes": "exception=RuntimeError('no reply read from the screen; exit None; timeout no PROBE-REPLY line on the screen; stderr: \\x1b[31mError: Something went wrong. Please try again.\\x1b[39m')" + }, + { + "entry": "cursor", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S5", + "arm": "S5/reload", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/cursor/tui/S5/S5_reload/run1.json" + ], + "notes": "no reload command for this entry" + }, + { + "entry": "cursor-userhooks", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S1", + "arm": "S1/native", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/cursor-userhooks/daemon/S1/S1_native/run1.json", + "out/cursor-userhooks/daemon/S1/S1_native/run2.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "cursor-userhooks", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S2", + "arm": "S2/hook-adds-skill", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 2, + "mechanism": "user ~/.cursor/hooks.json sessionStart", + "evidence": [ + "out/cursor-userhooks/daemon/S2/S2_hook-adds-skill/run1.json", + "out/cursor-userhooks/daemon/S2/S2_hook-adds-skill/run2.json" + ], + "notes": "hook never fired skill file absent after the turn stopReason=end_turn tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "cursor-userhooks", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S2", + "arm": "S2/hook-creates-root", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 2, + "mechanism": "user ~/.cursor/hooks.json sessionStart", + "evidence": [ + "out/cursor-userhooks/daemon/S2/S2_hook-creates-root/run1.json", + "out/cursor-userhooks/daemon/S2/S2_hook-creates-root/run2.json" + ], + "notes": "hook never fired skill file absent after the turn stopReason=end_turn tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "cursor-userhooks", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S2", + "arm": "S2/registration", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 1, + "mechanism": "user ~/.cursor/hooks.json workspaceOpen", + "evidence": [ + "out/cursor-userhooks/daemon/S2/S2_registration/run1.json" + ], + "notes": "skill file absent after the turn stopReason=end_turn tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "cursor-userhooks", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S1", + "arm": "S1/native", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/cursor-userhooks/print/S1/S1_native/run1.json", + "out/cursor-userhooks/print/S1/S1_native/run2.json" + ], + "notes": "tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "cursor-userhooks", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S2", + "arm": "S2/hook-adds-skill", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": "user ~/.cursor/hooks.json sessionStart", + "evidence": [ + "out/cursor-userhooks/print/S2/S2_hook-adds-skill/run1.json", + "out/cursor-userhooks/print/S2/S2_hook-adds-skill/run2.json" + ], + "notes": "tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "cursor-userhooks", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S2", + "arm": "S2/hook-creates-root", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": true, + "runs": 3, + "mechanism": "user ~/.cursor/hooks.json sessionStart", + "evidence": [ + "out/cursor-userhooks/print/S2/S2_hook-creates-root/run1.json", + "out/cursor-userhooks/print/S2/S2_hook-creates-root/run2.json", + "out/cursor-userhooks/print/S2/S2_hook-creates-root/run3.json" + ], + "notes": "tools_used=3 skill_reads=1 searches=3; tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "cursor-userhooks", + "harness": "cursor", + "version": "2026.09.15-d2fe57e", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S2", + "arm": "S2/registration", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "visible_after_reload", + "flaky": false, + "runs": 2, + "mechanism": "user ~/.cursor/hooks.json workspaceOpen", + "evidence": [ + "out/cursor-userhooks/print/S2/S2_registration/run1.json", + "out/cursor-userhooks/print/S2/S2_registration/run2.json" + ], + "notes": "tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "gemini", + "harness": "gemini", + "version": "0.60.0", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S0", + "arm": "S0/none", + "root": null, + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/gemini/daemon/S0/S0_none/run1.json", + "out/gemini/daemon/S0/S0_none/run2.json" + ], + "notes": "exception=RuntimeError('vendor exit -9 with no reply; session/new failed: {\"jsonrpc\": \"2.0\", \"id\": 2, \"error\": {\"code\": -32000, \"message\": \"This client is no longer supported for Gemini Code Assist for individuals. To continue using Gemini, please migrate to the Antigravity suite of products: https://antigravity.google\"}} tools_used=0; stderr: ')" + }, + { + "entry": "gemini", + "harness": "gemini", + "version": "0.60.0", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S1", + "arm": "S1/native", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/gemini/daemon/S1/S1_native/run1.json", + "out/gemini/daemon/S1/S1_native/run2.json" + ], + "notes": "exception=RuntimeError('vendor exit -9 with no reply; session/new failed: {\"jsonrpc\": \"2.0\", \"id\": 2, \"error\": {\"code\": -32000, \"message\": \"This client is no longer supported for Gemini Code Assist for individuals. To continue using Gemini, please migrate to the Antigravity suite of products: https://antigravity.google\"}} tools_used=0; stderr: ')" + }, + { + "entry": "gemini", + "harness": "gemini", + "version": "0.60.0", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S10", + "arm": "S10/hook-from-peer", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/gemini/daemon/S10/S10_hook-from-peer/run1.json", + "out/gemini/daemon/S10/S10_hook-from-peer/run2.json" + ], + "notes": "exception=RuntimeError('no reply read from the output; exit None; session/new failed: {\"jsonrpc\": \"2.0\", \"id\": 2, \"error\": {\"code\": -32000, \"message\": \"This client is no longer supported for Gemini Code Assist for individuals. To continue using Gemini, please migrate to the Antigravity suite of products: https://antigravity.google\"}} tools_used=0; stderr: '); exception=RuntimeError('no reply read from the output; exit None; session/new failed: {\"jsonrpc\": \"2.0\", \"id\": 2, \"error\": {\"code\": -32000, \"message\": \"Resource has been exhausted (e.g. check quota).\"}} tools_used=0; stderr: ')" + }, + { + "entry": "gemini", + "harness": "gemini", + "version": "0.60.0", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S2", + "arm": "S2/hook-adds-skill", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/gemini/daemon/S2/S2_hook-adds-skill/run1.json" + ], + "notes": "S1 failed" + }, + { + "entry": "gemini", + "harness": "gemini", + "version": "0.60.0", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S2", + "arm": "S2/hook-creates-root", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/gemini/daemon/S2/S2_hook-creates-root/run1.json" + ], + "notes": "S1 failed" + }, + { + "entry": "gemini", + "harness": "gemini", + "version": "0.60.0", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S2", + "arm": "S2/registration", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/gemini/daemon/S2/S2_registration/run1.json" + ], + "notes": "S1 failed" + }, + { + "entry": "gemini", + "harness": "gemini", + "version": "0.60.0", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S3", + "arm": "S3/gitignore", + "root": ".gemini/skills", + "exclusion": "gitignore", + "verdict": "untested", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/gemini/daemon/S3/S3_gitignore/run1.json" + ], + "notes": "S1 failed" + }, + { + "entry": "gemini", + "harness": "gemini", + "version": "0.60.0", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S3", + "arm": "S3/info-exclude", + "root": ".gemini/skills", + "exclusion": "info-exclude", + "verdict": "untested", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/gemini/daemon/S3/S3_info-exclude/run1.json" + ], + "notes": "S1 failed" + }, + { + "entry": "gemini", + "harness": "gemini", + "version": "0.60.0", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": null, + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/gemini/daemon/S4/S4_all-roots/run1.json" + ], + "notes": "S1 failed" + }, + { + "entry": "gemini", + "harness": "gemini", + "version": "0.60.0", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S5", + "arm": "S5/add", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/gemini/daemon/S5/S5_add/run1.json", + "out/gemini/daemon/S5/S5_add/run2.json" + ], + "notes": "exception=RuntimeError('no reply read from the output; exit None; session/new failed: {\"jsonrpc\": \"2.0\", \"id\": 2, \"error\": {\"code\": -32000, \"message\": \"This client is no longer supported for Gemini Code Assist for individuals. To continue using Gemini, please migrate to the Antigravity suite of products: https://antigravity.google\"}} tools_used=0; stderr: ')" + }, + { + "entry": "gemini", + "harness": "gemini", + "version": "0.60.0", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S5", + "arm": "S5/delete", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/gemini/daemon/S5/S5_delete/run1.json", + "out/gemini/daemon/S5/S5_delete/run2.json" + ], + "notes": "exception=RuntimeError('no reply read from the output; exit None; session/new failed: {\"jsonrpc\": \"2.0\", \"id\": 2, \"error\": {\"code\": -32000, \"message\": \"Resource has been exhausted (e.g. check quota).\"}} tools_used=0; stderr: ')" + }, + { + "entry": "gemini", + "harness": "gemini", + "version": "0.60.0", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S5", + "arm": "S5/update", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/gemini/daemon/S5/S5_update/run1.json", + "out/gemini/daemon/S5/S5_update/run2.json" + ], + "notes": "exception=RuntimeError('no reply read from the output; exit None; session/new failed: {\"jsonrpc\": \"2.0\", \"id\": 2, \"error\": {\"code\": -32000, \"message\": \"This client is no longer supported for Gemini Code Assist for individuals. To continue using Gemini, please migrate to the Antigravity suite of products: https://antigravity.google\"}} tools_used=0; stderr: '); exception=RuntimeError('no reply read from the output; exit None; session/new failed: {\"jsonrpc\": \"2.0\", \"id\": 2, \"error\": {\"code\": -32000, \"message\": \"Resource has been exhausted (e.g. check quota).\"}} tools_used=0; stderr: ')" + }, + { + "entry": "gemini", + "harness": "gemini", + "version": "0.60.0", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S6", + "arm": "S6/delete", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/gemini/daemon/S6/S6_delete/run1.json", + "out/gemini/daemon/S6/S6_delete/run2.json" + ], + "notes": "exception=RuntimeError('no reply read from the output; exit None; session/new failed: {\"jsonrpc\": \"2.0\", \"id\": 2, \"error\": {\"code\": -32000, \"message\": \"This client is no longer supported for Gemini Code Assist for individuals. To continue using Gemini, please migrate to the Antigravity suite of products: https://antigravity.google\"}} tools_used=0; stderr: ')" + }, + { + "entry": "gemini", + "harness": "gemini", + "version": "0.60.0", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S6", + "arm": "S6/update", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/gemini/daemon/S6/S6_update/run1.json", + "out/gemini/daemon/S6/S6_update/run2.json" + ], + "notes": "exception=RuntimeError('no reply read from the output; exit None; session/new failed: {\"jsonrpc\": \"2.0\", \"id\": 2, \"error\": {\"code\": -32000, \"message\": \"Resource has been exhausted (e.g. check quota).\"}} tools_used=0; stderr: ')" + }, + { + "entry": "gemini", + "harness": "gemini", + "version": "0.60.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S0", + "arm": "S0/none", + "root": null, + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/gemini/print/S0/S0_none/run1.json", + "out/gemini/print/S0/S0_none/run2.json" + ], + "notes": "exception=RuntimeError(\"vendor exit 1 with no reply; extract failed: JSONDecodeError('Expecting value: line 1 column 1 (char 0)'); stderr: ctIdError (file:///opt/homebrew/lib/node_modules/@google/gemini-cli/bundle/chunk-M6NSK26M.js:310656:11) | at _doSetupUser (file:///opt/homebrew/lib/node_modules/@google/gemini-cli/bundle/chunk-M6NSK26M.js:310645:5) | at process.processTicksAndRejections (node:internal/process/task_queues:104:5) | \")" + }, + { + "entry": "gemini", + "harness": "gemini", + "version": "0.60.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S1", + "arm": "S1/native", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/gemini/print/S1/S1_native/run1.json", + "out/gemini/print/S1/S1_native/run2.json" + ], + "notes": "exception=RuntimeError(\"vendor exit 1 with no reply; extract failed: JSONDecodeError('Expecting value: line 1 column 1 (char 0)'); stderr: ctIdError (file:///opt/homebrew/lib/node_modules/@google/gemini-cli/bundle/chunk-M6NSK26M.js:310656:11) | at _doSetupUser (file:///opt/homebrew/lib/node_modules/@google/gemini-cli/bundle/chunk-M6NSK26M.js:310645:5) | at process.processTicksAndRejections (node:internal/process/task_queues:104:5) | \")" + }, + { + "entry": "gemini", + "harness": "gemini", + "version": "0.60.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S2", + "arm": "S2/hook-adds-skill", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/gemini/print/S2/S2_hook-adds-skill/run1.json" + ], + "notes": "S1 failed" + }, + { + "entry": "gemini", + "harness": "gemini", + "version": "0.60.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S2", + "arm": "S2/hook-creates-root", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/gemini/print/S2/S2_hook-creates-root/run1.json" + ], + "notes": "S1 failed" + }, + { + "entry": "gemini", + "harness": "gemini", + "version": "0.60.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S2", + "arm": "S2/registration", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/gemini/print/S2/S2_registration/run1.json" + ], + "notes": "S1 failed" + }, + { + "entry": "gemini", + "harness": "gemini", + "version": "0.60.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S3", + "arm": "S3/gitignore", + "root": ".gemini/skills", + "exclusion": "gitignore", + "verdict": "untested", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/gemini/print/S3/S3_gitignore/run1.json" + ], + "notes": "S1 failed" + }, + { + "entry": "gemini", + "harness": "gemini", + "version": "0.60.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S3", + "arm": "S3/info-exclude", + "root": ".gemini/skills", + "exclusion": "info-exclude", + "verdict": "untested", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/gemini/print/S3/S3_info-exclude/run1.json" + ], + "notes": "S1 failed" + }, + { + "entry": "gemini", + "harness": "gemini", + "version": "0.60.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": null, + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/gemini/print/S4/S4_all-roots/run1.json" + ], + "notes": "S1 failed" + }, + { + "entry": "gemini", + "harness": "gemini", + "version": "0.60.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S6", + "arm": "S6/delete", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/gemini/print/S6/S6_delete/run1.json", + "out/gemini/print/S6/S6_delete/run2.json" + ], + "notes": "exception=RuntimeError(\"no reply read from the output; exit 1; extract failed: JSONDecodeError('Expecting value: line 1 column 1 (char 0)'); stderr: Error authenticating: IneligibleTierError: This client is no longer supported for Gemini Code Assist for individuals. To continue using Gemini, please migrate to the Antigravity suite of products: https://antigravity.google\")" + }, + { + "entry": "gemini", + "harness": "gemini", + "version": "0.60.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S6", + "arm": "S6/update", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/gemini/print/S6/S6_update/run1.json", + "out/gemini/print/S6/S6_update/run2.json" + ], + "notes": "exception=RuntimeError(\"no reply read from the output; exit 1; extract failed: JSONDecodeError('Expecting value: line 1 column 1 (char 0)'); stderr: Error authenticating: IneligibleTierError: This client is no longer supported for Gemini Code Assist for individuals. To continue using Gemini, please migrate to the Antigravity suite of products: https://antigravity.google\")" + }, + { + "entry": "gemini", + "harness": "gemini", + "version": "0.60.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S7", + "arm": "S7/add", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/gemini/print/S7/S7_add/run1.json", + "out/gemini/print/S7/S7_add/run2.json" + ], + "notes": "exception=RuntimeError(\"no reply read from the output; exit 1; extract failed: JSONDecodeError('Expecting value: line 1 column 1 (char 0)'); stderr: Error authenticating: _GaxiosError: Resource has been exhausted (e.g. check quota).\")" + }, + { + "entry": "gemini", + "harness": "gemini", + "version": "0.60.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S7", + "arm": "S7/update", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/gemini/print/S7/S7_update/run1.json", + "out/gemini/print/S7/S7_update/run2.json" + ], + "notes": "exception=RuntimeError(\"no reply read from the output; exit 1; extract failed: JSONDecodeError('Expecting value: line 1 column 1 (char 0)'); stderr: Error authenticating: _GaxiosError: Resource has been exhausted (e.g. check quota).\")" + }, + { + "entry": "gemini", + "harness": "gemini", + "version": "0.60.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S8", + "arm": "S8/ancestor", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/gemini/print/S8/S8_ancestor/run1.json", + "out/gemini/print/S8/S8_ancestor/run2.json" + ], + "notes": "exception=RuntimeError(\"no reply read from the output; exit 1; extract failed: JSONDecodeError('Expecting value: line 1 column 1 (char 0)'); stderr: Error authenticating: _GaxiosError: Resource has been exhausted (e.g. check quota).\")" + }, + { + "entry": "gemini", + "harness": "gemini", + "version": "0.60.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S8", + "arm": "S8/local", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/gemini/print/S8/S8_local/run1.json", + "out/gemini/print/S8/S8_local/run2.json" + ], + "notes": "exception=RuntimeError(\"no reply read from the output; exit 1; extract failed: JSONDecodeError('Expecting value: line 1 column 1 (char 0)'); stderr: Error authenticating: _GaxiosError: Resource has been exhausted (e.g. check quota).\")" + }, + { + "entry": "gemini", + "harness": "gemini", + "version": "0.60.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S9", + "arm": "S9/linked-other", + "root": ".gemini/skills", + "exclusion": "info-exclude", + "verdict": "untested", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/gemini/print/S9/S9_linked-other/run1.json", + "out/gemini/print/S9/S9_linked-other/run2.json" + ], + "notes": "exception=RuntimeError(\"no reply read from the output; exit 1; extract failed: JSONDecodeError('Expecting value: line 1 column 1 (char 0)'); stderr: Error authenticating: _GaxiosError: Resource has been exhausted (e.g. check quota).\")" + }, + { + "entry": "gemini", + "harness": "gemini", + "version": "0.60.0", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S9", + "arm": "S9/linked-own", + "root": ".gemini/skills", + "exclusion": "info-exclude", + "verdict": "untested", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/gemini/print/S9/S9_linked-own/run1.json", + "out/gemini/print/S9/S9_linked-own/run2.json" + ], + "notes": "exception=RuntimeError(\"no reply read from the output; exit 1; extract failed: JSONDecodeError('Expecting value: line 1 column 1 (char 0)'); stderr: Error authenticating: _GaxiosError: Resource has been exhausted (e.g. check quota).\")" + }, + { + "entry": "gemini", + "harness": "gemini", + "version": "0.60.0", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S0", + "arm": "S0/none", + "root": null, + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/gemini/tui/S0/S0_none/run1.json", + "out/gemini/tui/S0/S0_none/run2.json" + ], + "notes": "exception=RuntimeError('no reply read from the screen; exit None; timeout no PROBE-REPLY line on the screen; stderr: \\x1b[38;5;194m\u2502\\x1b[39m | \\x1b[38;5;194m\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\\x1b[39')" + }, + { + "entry": "gemini", + "harness": "gemini", + "version": "0.60.0", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S1", + "arm": "S1/native", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/gemini/tui/S1/S1_native/run1.json", + "out/gemini/tui/S1/S1_native/run2.json" + ], + "notes": "exception=RuntimeError('no reply read from the screen; exit None; timeout no PROBE-REPLY line on the screen; stderr: \\x1b[38;5;194m\u2502\\x1b[39m | \\x1b[38;5;194m\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\\x1b[39')" + }, + { + "entry": "gemini", + "harness": "gemini", + "version": "0.60.0", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S2", + "arm": "S2/hook-adds-skill", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/gemini/tui/S2/S2_hook-adds-skill/run1.json" + ], + "notes": "S1 failed" + }, + { + "entry": "gemini", + "harness": "gemini", + "version": "0.60.0", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S5", + "arm": "S5/add", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/gemini/tui/S5/S5_add/run1.json" + ], + "notes": "S1 failed" + }, + { + "entry": "gemini", + "harness": "gemini", + "version": "0.60.0", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S5", + "arm": "S5/reload", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/gemini/tui/S5/S5_reload/run1.json" + ], + "notes": "S1 failed" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S0", + "arm": "S0/none", + "root": null, + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/kiro/daemon/S0/S0_none/run1.json", + "out/kiro/daemon/S0/S0_none/run2.json" + ], + "notes": "stopReason=end_turn tools_used=0; stopReason=end_turn tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S1", + "arm": "S1/native", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/kiro/daemon/S1/S1_native/run1.json", + "out/kiro/daemon/S1/S1_native/run2.json" + ], + "notes": "stopReason=end_turn tools_used=1; stopReason=end_turn tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S10", + "arm": "S10/hook-from-peer", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "visible_live", + "flaky": false, + "runs": 2, + "mechanism": "agent hooks.agentSpawn (cli 2.x)", + "evidence": [ + "out/kiro/daemon/S10/S10_hook-from-peer/run1.json", + "out/kiro/daemon/S10/S10_hook-from-peer/run2.json" + ], + "notes": "turn1=not_visible peer=visible_first_turn stopReason=end_turn tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S2", + "arm": "S2/hook-adds-skill", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": "agent hooks.agentSpawn (cli 2.x)", + "evidence": [ + "out/kiro/daemon/S2/S2_hook-adds-skill/run1.json", + "out/kiro/daemon/S2/S2_hook-adds-skill/run2.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S2", + "arm": "S2/hook-creates-root", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": "agent hooks.agentSpawn (cli 2.x)", + "evidence": [ + "out/kiro/daemon/S2/S2_hook-creates-root/run1.json", + "out/kiro/daemon/S2/S2_hook-creates-root/run2.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S2", + "arm": "S2/registration", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/kiro/daemon/S2/S2_registration/run1.json" + ], + "notes": "no registration mechanism for this entry" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S3", + "arm": "S3/gitignore", + "root": ".kiro/skills", + "exclusion": "gitignore", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/kiro/daemon/S3/S3_gitignore/run1.json", + "out/kiro/daemon/S3/S3_gitignore/run2.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S3", + "arm": "S3/info-exclude", + "root": ".kiro/skills", + "exclusion": "info-exclude", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/kiro/daemon/S3/S3_info-exclude/run1.json", + "out/kiro/daemon/S3/S3_info-exclude/run2.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".agent/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/kiro/daemon/S4/S4_all-roots/run10.json", + "out/kiro/daemon/S4/S4_all-roots/run20.json" + ], + "notes": "found=['kiro'] leaked=[] stopReason=end_turn tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/kiro/daemon/S4/S4_all-roots/run12.json", + "out/kiro/daemon/S4/S4_all-roots/run2.json" + ], + "notes": "found=['kiro'] leaked=[] stopReason=end_turn tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".claude/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/kiro/daemon/S4/S4_all-roots/run1.json", + "out/kiro/daemon/S4/S4_all-roots/run11.json" + ], + "notes": "found=['kiro'] leaked=[] stopReason=end_turn tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".codex/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/kiro/daemon/S4/S4_all-roots/run13.json", + "out/kiro/daemon/S4/S4_all-roots/run3.json" + ], + "notes": "found=['kiro'] leaked=[] stopReason=end_turn tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/kiro/daemon/S4/S4_all-roots/run14.json", + "out/kiro/daemon/S4/S4_all-roots/run4.json" + ], + "notes": "found=['kiro'] leaked=[] stopReason=end_turn tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/kiro/daemon/S4/S4_all-roots/run16.json", + "out/kiro/daemon/S4/S4_all-roots/run6.json" + ], + "notes": "found=['kiro'] leaked=[] stopReason=end_turn tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".github/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/kiro/daemon/S4/S4_all-roots/run15.json", + "out/kiro/daemon/S4/S4_all-roots/run5.json" + ], + "notes": "found=['kiro'] leaked=[] stopReason=end_turn tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/kiro/daemon/S4/S4_all-roots/run17.json", + "out/kiro/daemon/S4/S4_all-roots/run7.json" + ], + "notes": "found=['kiro'] leaked=[] stopReason=end_turn tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/kiro/daemon/S4/S4_all-roots/run19.json", + "out/kiro/daemon/S4/S4_all-roots/run9.json" + ], + "notes": "found=['kiro'] leaked=[] stopReason=end_turn tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/kiro/daemon/S4/S4_all-roots/run18.json", + "out/kiro/daemon/S4/S4_all-roots/run8.json" + ], + "notes": "found=['kiro'] leaked=[] stopReason=end_turn tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-agent", + "root": ".agent/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/kiro/daemon/S4/S4_confirm-agent/run1.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-agents", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/kiro/daemon/S4/S4_confirm-agents/run1.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-claude", + "root": ".claude/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/kiro/daemon/S4/S4_confirm-claude/run1.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-codex", + "root": ".codex/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/kiro/daemon/S4/S4_confirm-codex/run1.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-cursor", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/kiro/daemon/S4/S4_confirm-cursor/run1.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-gemini", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/kiro/daemon/S4/S4_confirm-gemini/run1.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-github", + "root": ".github/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/kiro/daemon/S4/S4_confirm-github/run1.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-opencode", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/kiro/daemon/S4/S4_confirm-opencode/run1.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-pi", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/kiro/daemon/S4/S4_confirm-pi/run1.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S5", + "arm": "S5/add", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "visible_live", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/kiro/daemon/S5/S5_add/run1.json", + "out/kiro/daemon/S5/S5_add/run2.json" + ], + "notes": "turn1=not_visible stopReason=end_turn tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S5", + "arm": "S5/delete", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "revoked", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/kiro/daemon/S5/S5_delete/run1.json", + "out/kiro/daemon/S5/S5_delete/run2.json" + ], + "notes": "turn1=visible_first_turn stopReason=end_turn tools_used=1 skill_reads=0 searches=0" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S5", + "arm": "S5/update", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "visible_live", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/kiro/daemon/S5/S5_update/run1.json", + "out/kiro/daemon/S5/S5_update/run2.json" + ], + "notes": "turn1=visible_first_turn stopReason=end_turn tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S6", + "arm": "S6/delete", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "revoked", + "flaky": false, + "runs": 2, + "mechanism": "agent hooks.agentSpawn (cli 2.x)", + "evidence": [ + "out/kiro/daemon/S6/S6_delete/run1.json", + "out/kiro/daemon/S6/S6_delete/run2.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S6", + "arm": "S6/update", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": "agent hooks.agentSpawn (cli 2.x)", + "evidence": [ + "out/kiro/daemon/S6/S6_update/run1.json", + "out/kiro/daemon/S6/S6_update/run2.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S0", + "arm": "S0/none", + "root": null, + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/kiro/print/S0/S0_none/run1.json", + "out/kiro/print/S0/S0_none/run2.json" + ], + "notes": "" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S1", + "arm": "S1/native", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/kiro/print/S1/S1_native/run1.json", + "out/kiro/print/S1/S1_native/run2.json" + ], + "notes": "" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S2", + "arm": "S2/hook-adds-skill", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": "agent hooks.agentSpawn (cli 2.x)", + "evidence": [ + "out/kiro/print/S2/S2_hook-adds-skill/run1.json", + "out/kiro/print/S2/S2_hook-adds-skill/run2.json" + ], + "notes": "tools=unobserved" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S2", + "arm": "S2/hook-creates-root", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": "agent hooks.agentSpawn (cli 2.x)", + "evidence": [ + "out/kiro/print/S2/S2_hook-creates-root/run1.json", + "out/kiro/print/S2/S2_hook-creates-root/run2.json" + ], + "notes": "tools=unobserved" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S2", + "arm": "S2/registration", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/kiro/print/S2/S2_registration/run1.json" + ], + "notes": "no registration mechanism for this entry" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S3", + "arm": "S3/gitignore", + "root": ".kiro/skills", + "exclusion": "gitignore", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/kiro/print/S3/S3_gitignore/run1.json", + "out/kiro/print/S3/S3_gitignore/run2.json" + ], + "notes": "" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S3", + "arm": "S3/info-exclude", + "root": ".kiro/skills", + "exclusion": "info-exclude", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/kiro/print/S3/S3_info-exclude/run1.json", + "out/kiro/print/S3/S3_info-exclude/run2.json" + ], + "notes": "" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".agent/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/kiro/print/S4/S4_all-roots/run10.json", + "out/kiro/print/S4/S4_all-roots/run20.json" + ], + "notes": "found=['kiro'] leaked=[]" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/kiro/print/S4/S4_all-roots/run12.json", + "out/kiro/print/S4/S4_all-roots/run2.json" + ], + "notes": "found=['kiro'] leaked=[]" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".claude/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/kiro/print/S4/S4_all-roots/run1.json", + "out/kiro/print/S4/S4_all-roots/run11.json" + ], + "notes": "found=['kiro'] leaked=[]" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".codex/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/kiro/print/S4/S4_all-roots/run13.json", + "out/kiro/print/S4/S4_all-roots/run3.json" + ], + "notes": "found=['kiro'] leaked=[]" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/kiro/print/S4/S4_all-roots/run14.json", + "out/kiro/print/S4/S4_all-roots/run4.json" + ], + "notes": "found=['kiro'] leaked=[]" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/kiro/print/S4/S4_all-roots/run16.json", + "out/kiro/print/S4/S4_all-roots/run6.json" + ], + "notes": "found=['kiro'] leaked=[]" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".github/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/kiro/print/S4/S4_all-roots/run15.json", + "out/kiro/print/S4/S4_all-roots/run5.json" + ], + "notes": "found=['kiro'] leaked=[]" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/kiro/print/S4/S4_all-roots/run17.json", + "out/kiro/print/S4/S4_all-roots/run7.json" + ], + "notes": "found=['kiro'] leaked=[]" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/kiro/print/S4/S4_all-roots/run19.json", + "out/kiro/print/S4/S4_all-roots/run9.json" + ], + "notes": "found=['kiro'] leaked=[]" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/kiro/print/S4/S4_all-roots/run18.json", + "out/kiro/print/S4/S4_all-roots/run8.json" + ], + "notes": "found=['kiro'] leaked=[]" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-agent", + "root": ".agent/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/kiro/print/S4/S4_confirm-agent/run1.json" + ], + "notes": "" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-agents", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/kiro/print/S4/S4_confirm-agents/run1.json" + ], + "notes": "" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-claude", + "root": ".claude/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/kiro/print/S4/S4_confirm-claude/run1.json" + ], + "notes": "" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-codex", + "root": ".codex/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/kiro/print/S4/S4_confirm-codex/run1.json" + ], + "notes": "" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-cursor", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/kiro/print/S4/S4_confirm-cursor/run1.json" + ], + "notes": "" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-gemini", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/kiro/print/S4/S4_confirm-gemini/run1.json" + ], + "notes": "" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-github", + "root": ".github/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/kiro/print/S4/S4_confirm-github/run1.json" + ], + "notes": "" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-opencode", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/kiro/print/S4/S4_confirm-opencode/run1.json" + ], + "notes": "" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-pi", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/kiro/print/S4/S4_confirm-pi/run1.json" + ], + "notes": "" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S6", + "arm": "S6/delete", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "revoked", + "flaky": false, + "runs": 2, + "mechanism": "agent hooks.agentSpawn (cli 2.x)", + "evidence": [ + "out/kiro/print/S6/S6_delete/run1.json", + "out/kiro/print/S6/S6_delete/run2.json" + ], + "notes": "tools=unobserved" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S6", + "arm": "S6/update", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": "agent hooks.agentSpawn (cli 2.x)", + "evidence": [ + "out/kiro/print/S6/S6_update/run1.json", + "out/kiro/print/S6/S6_update/run2.json" + ], + "notes": "tools=unobserved" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S7", + "arm": "S7/add", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/kiro/print/S7/S7_add/run1.json", + "out/kiro/print/S7/S7_add/run2.json" + ], + "notes": "turn1=not_visible session=most-recent-in-cwd tools=unobserved" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S7", + "arm": "S7/update", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/kiro/print/S7/S7_update/run1.json", + "out/kiro/print/S7/S7_update/run2.json" + ], + "notes": "turn1=visible_first_turn session=most-recent-in-cwd tools=unobserved" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S8", + "arm": "S8/ancestor", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/kiro/print/S8/S8_ancestor/run1.json", + "out/kiro/print/S8/S8_ancestor/run2.json" + ], + "notes": "cwd=sub/dir tools=unobserved" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S8", + "arm": "S8/local", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/kiro/print/S8/S8_local/run1.json", + "out/kiro/print/S8/S8_local/run2.json" + ], + "notes": "cwd=sub/dir tools=unobserved" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S9", + "arm": "S9/linked-other", + "root": ".kiro/skills", + "exclusion": "info-exclude", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/kiro/print/S9/S9_linked-other/run1.json", + "out/kiro/print/S9/S9_linked-other/run2.json" + ], + "notes": "cwd=wt-b skill_in=main tools=unobserved" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S9", + "arm": "S9/linked-own", + "root": ".kiro/skills", + "exclusion": "info-exclude", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/kiro/print/S9/S9_linked-own/run1.json", + "out/kiro/print/S9/S9_linked-own/run2.json" + ], + "notes": "cwd=wt-b skill_in=linked tools=unobserved" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S0", + "arm": "S0/none", + "root": null, + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/kiro/tui/S0/S0_none/run1.json", + "out/kiro/tui/S0/S0_none/run2.json" + ], + "notes": "dialog=Yes, I accept" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S1", + "arm": "S1/native", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/kiro/tui/S1/S1_native/run1.json", + "out/kiro/tui/S1/S1_native/run2.json" + ], + "notes": "dialog=Yes, I accept" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S2", + "arm": "S2/hook-adds-skill", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": "agent hooks.agentSpawn (cli 2.x)", + "evidence": [ + "out/kiro/tui/S2/S2_hook-adds-skill/run1.json", + "out/kiro/tui/S2/S2_hook-adds-skill/run2.json" + ], + "notes": "dialog=Yes, I accept" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S5", + "arm": "S5/add", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "visible_live", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/kiro/tui/S5/S5_add/run1.json", + "out/kiro/tui/S5/S5_add/run2.json" + ], + "notes": "turn1=not_visible" + }, + { + "entry": "kiro", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S5", + "arm": "S5/reload", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/kiro/tui/S5/S5_reload/run1.json" + ], + "notes": "no reload command for this entry" + }, + { + "entry": "kiro-agent-bare", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S0", + "arm": "S0/none", + "root": null, + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/kiro-agent-bare/daemon/S0/S0_none/run1.json", + "out/kiro-agent-bare/daemon/S0/S0_none/run2.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "kiro-agent-bare", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S1", + "arm": "S1/native", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/kiro-agent-bare/daemon/S1/S1_native/run1.json", + "out/kiro-agent-bare/daemon/S1/S1_native/run2.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "kiro-agent-bare", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S0", + "arm": "S0/none", + "root": null, + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/kiro-agent-bare/print/S0/S0_none/run1.json", + "out/kiro-agent-bare/print/S0/S0_none/run2.json" + ], + "notes": "tools=unobserved" + }, + { + "entry": "kiro-agent-bare", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S1", + "arm": "S1/native", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/kiro-agent-bare/print/S1/S1_native/run1.json", + "out/kiro-agent-bare/print/S1/S1_native/run2.json" + ], + "notes": "tools=unobserved" + }, + { + "entry": "kiro-agent-skills", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S0", + "arm": "S0/none", + "root": null, + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/kiro-agent-skills/daemon/S0/S0_none/run1.json", + "out/kiro-agent-skills/daemon/S0/S0_none/run2.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "kiro-agent-skills", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S1", + "arm": "S1/native", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/kiro-agent-skills/daemon/S1/S1_native/run1.json", + "out/kiro-agent-skills/daemon/S1/S1_native/run2.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "kiro-agent-skills", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S0", + "arm": "S0/none", + "root": null, + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/kiro-agent-skills/print/S0/S0_none/run1.json", + "out/kiro-agent-skills/print/S0/S0_none/run2.json" + ], + "notes": "tools=unobserved" + }, + { + "entry": "kiro-agent-skills", + "harness": "kiro", + "version": "kiro-cli 2.21.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S1", + "arm": "S1/native", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/kiro-agent-skills/print/S1/S1_native/run1.json", + "out/kiro-agent-skills/print/S1/S1_native/run2.json" + ], + "notes": "tools=unobserved" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S0", + "arm": "S0/none", + "root": null, + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v1/daemon/S0/S0_none/run1.json", + "out/opencode-v1/daemon/S0/S0_none/run2.json" + ], + "notes": "stopReason=end_turn tools_used=0; stopReason=end_turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S1", + "arm": "S1/native", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v1/daemon/S1/S1_native/run1.json", + "out/opencode-v1/daemon/S1/S1_native/run2.json" + ], + "notes": "stopReason=end_turn tools_used=1; stopReason=end_turn tools_used=0 skill_loads=1 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S10", + "arm": "S10/hook-from-peer", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": "plugin event session.created", + "evidence": [ + "out/opencode-v1/daemon/S10/S10_hook-from-peer/run1.json", + "out/opencode-v1/daemon/S10/S10_hook-from-peer/run2.json" + ], + "notes": "turn1=not_visible peer=not_visible stopReason=end_turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S2", + "arm": "S2/hook-adds-skill", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": "plugin event session.created", + "evidence": [ + "out/opencode-v1/daemon/S2/S2_hook-adds-skill/run1.json", + "out/opencode-v1/daemon/S2/S2_hook-adds-skill/run2.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S2", + "arm": "S2/hook-creates-root", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": true, + "runs": 3, + "mechanism": "plugin event session.created", + "evidence": [ + "out/opencode-v1/daemon/S2/S2_hook-creates-root/run1.json", + "out/opencode-v1/daemon/S2/S2_hook-creates-root/run2.json", + "out/opencode-v1/daemon/S2/S2_hook-creates-root/run3.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_loads=0 searches=0; exception=RuntimeError('vendor exit -15 with no reply; exception=TimeoutError() tools_used=0 skill_loads=0 searches=0; stderr: ')" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S2", + "arm": "S2/registration", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": "plugin experimental.chat.system.transform", + "evidence": [ + "out/opencode-v1/daemon/S2/S2_registration/run1.json", + "out/opencode-v1/daemon/S2/S2_registration/run2.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S3", + "arm": "S3/gitignore", + "root": ".opencode/skills", + "exclusion": "gitignore", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v1/daemon/S3/S3_gitignore/run1.json", + "out/opencode-v1/daemon/S3/S3_gitignore/run2.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_loads=1 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S3", + "arm": "S3/info-exclude", + "root": ".opencode/skills", + "exclusion": "info-exclude", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v1/daemon/S3/S3_info-exclude/run1.json", + "out/opencode-v1/daemon/S3/S3_info-exclude/run2.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_loads=1 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".agent/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v1/daemon/S4/S4_all-roots/run10.json", + "out/opencode-v1/daemon/S4/S4_all-roots/run20.json" + ], + "notes": "found=['agents', 'claude', 'opencode'] leaked=[] stopReason=end_turn tools_used=3 skill_loads=0 searches=3" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v1/daemon/S4/S4_all-roots/run12.json", + "out/opencode-v1/daemon/S4/S4_all-roots/run2.json" + ], + "notes": "found=['agents', 'claude', 'opencode'] leaked=[] stopReason=end_turn tools_used=3 skill_loads=0 searches=3" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".claude/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v1/daemon/S4/S4_all-roots/run1.json", + "out/opencode-v1/daemon/S4/S4_all-roots/run11.json" + ], + "notes": "found=['agents', 'claude', 'opencode'] leaked=[] stopReason=end_turn tools_used=3 skill_loads=0 searches=3" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".codex/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v1/daemon/S4/S4_all-roots/run13.json", + "out/opencode-v1/daemon/S4/S4_all-roots/run3.json" + ], + "notes": "found=['agents', 'claude', 'opencode'] leaked=[] stopReason=end_turn tools_used=3 skill_loads=0 searches=3" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v1/daemon/S4/S4_all-roots/run14.json", + "out/opencode-v1/daemon/S4/S4_all-roots/run4.json" + ], + "notes": "found=['agents', 'claude', 'opencode'] leaked=[] stopReason=end_turn tools_used=3 skill_loads=0 searches=3" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v1/daemon/S4/S4_all-roots/run16.json", + "out/opencode-v1/daemon/S4/S4_all-roots/run6.json" + ], + "notes": "found=['agents', 'claude', 'opencode'] leaked=[] stopReason=end_turn tools_used=3 skill_loads=0 searches=3" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".github/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v1/daemon/S4/S4_all-roots/run15.json", + "out/opencode-v1/daemon/S4/S4_all-roots/run5.json" + ], + "notes": "found=['agents', 'claude', 'opencode'] leaked=[] stopReason=end_turn tools_used=3 skill_loads=0 searches=3" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v1/daemon/S4/S4_all-roots/run17.json", + "out/opencode-v1/daemon/S4/S4_all-roots/run7.json" + ], + "notes": "found=['agents', 'claude', 'opencode'] leaked=[] stopReason=end_turn tools_used=3 skill_loads=0 searches=3" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v1/daemon/S4/S4_all-roots/run19.json", + "out/opencode-v1/daemon/S4/S4_all-roots/run9.json" + ], + "notes": "found=['agents', 'claude', 'opencode'] leaked=[] stopReason=end_turn tools_used=3 skill_loads=0 searches=3" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v1/daemon/S4/S4_all-roots/run18.json", + "out/opencode-v1/daemon/S4/S4_all-roots/run8.json" + ], + "notes": "found=['agents', 'claude', 'opencode'] leaked=[] stopReason=end_turn tools_used=3 skill_loads=0 searches=3" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-agent", + "root": ".agent/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/opencode-v1/daemon/S4/S4_confirm-agent/run1.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-codex", + "root": ".codex/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/opencode-v1/daemon/S4/S4_confirm-codex/run1.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-cursor", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/opencode-v1/daemon/S4/S4_confirm-cursor/run1.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-gemini", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/opencode-v1/daemon/S4/S4_confirm-gemini/run1.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-github", + "root": ".github/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/opencode-v1/daemon/S4/S4_confirm-github/run1.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-kiro", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/opencode-v1/daemon/S4/S4_confirm-kiro/run1.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-pi", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/opencode-v1/daemon/S4/S4_confirm-pi/run1.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S5", + "arm": "S5/add", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v1/daemon/S5/S5_add/run1.json", + "out/opencode-v1/daemon/S5/S5_add/run2.json" + ], + "notes": "turn1=not_visible stopReason=end_turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S5", + "arm": "S5/delete", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "stale", + "flaky": true, + "runs": 3, + "mechanism": null, + "evidence": [ + "out/opencode-v1/daemon/S5/S5_delete/run1.json", + "out/opencode-v1/daemon/S5/S5_delete/run2.json", + "out/opencode-v1/daemon/S5/S5_delete/run3.json" + ], + "notes": "turn1=visible_first_turn stopReason=end_turn tools_used=1 skill_loads=2 searches=1; turn1=visible_first_turn stopReason=end_turn tools_used=1 skill_loads=1 searches=1; turn1=visible_first_turn stopReason=end_turn tools_used=2 skill_loads=2 searches=2" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S5", + "arm": "S5/update", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "stale", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v1/daemon/S5/S5_update/run1.json", + "out/opencode-v1/daemon/S5/S5_update/run2.json" + ], + "notes": "turn1=visible_first_turn stopReason=end_turn tools_used=0 skill_loads=1 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S6", + "arm": "S6/delete", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "revoked", + "flaky": false, + "runs": 2, + "mechanism": "plugin event session.created", + "evidence": [ + "out/opencode-v1/daemon/S6/S6_delete/run1.json", + "out/opencode-v1/daemon/S6/S6_delete/run2.json" + ], + "notes": "stopReason=end_turn tools_used=3 skill_loads=1 searches=3; stopReason=end_turn tools_used=2 skill_loads=1 searches=2" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S6", + "arm": "S6/update", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "stale", + "flaky": false, + "runs": 2, + "mechanism": "plugin event session.created", + "evidence": [ + "out/opencode-v1/daemon/S6/S6_update/run1.json", + "out/opencode-v1/daemon/S6/S6_update/run2.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_loads=1 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S0", + "arm": "S0/none", + "root": null, + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v1/print/S0/S0_none/run1.json", + "out/opencode-v1/print/S0/S0_none/run2.json" + ], + "notes": "tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S1", + "arm": "S1/native", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v1/print/S1/S1_native/run1.json", + "out/opencode-v1/print/S1/S1_native/run2.json" + ], + "notes": "tools_used=0 skill_loads=1 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S2", + "arm": "S2/hook-adds-skill", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": "plugin event session.created", + "evidence": [ + "out/opencode-v1/print/S2/S2_hook-adds-skill/run1.json", + "out/opencode-v1/print/S2/S2_hook-adds-skill/run2.json" + ], + "notes": "tools_used=0 skill_loads=1 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S2", + "arm": "S2/hook-creates-root", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": "plugin event session.created", + "evidence": [ + "out/opencode-v1/print/S2/S2_hook-creates-root/run1.json", + "out/opencode-v1/print/S2/S2_hook-creates-root/run2.json" + ], + "notes": "tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S2", + "arm": "S2/registration", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": "plugin experimental.chat.system.transform", + "evidence": [ + "out/opencode-v1/print/S2/S2_registration/run1.json", + "out/opencode-v1/print/S2/S2_registration/run2.json" + ], + "notes": "tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S3", + "arm": "S3/gitignore", + "root": ".opencode/skills", + "exclusion": "gitignore", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v1/print/S3/S3_gitignore/run1.json", + "out/opencode-v1/print/S3/S3_gitignore/run2.json" + ], + "notes": "tools_used=0 skill_loads=1 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S3", + "arm": "S3/info-exclude", + "root": ".opencode/skills", + "exclusion": "info-exclude", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v1/print/S3/S3_info-exclude/run1.json", + "out/opencode-v1/print/S3/S3_info-exclude/run2.json" + ], + "notes": "tools_used=0 skill_loads=1 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".agent/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v1/print/S4/S4_all-roots/run10.json", + "out/opencode-v1/print/S4/S4_all-roots/run20.json" + ], + "notes": "found=['agents', 'claude', 'opencode'] leaked=[] tools_used=0 skill_loads=3 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v1/print/S4/S4_all-roots/run12.json", + "out/opencode-v1/print/S4/S4_all-roots/run2.json" + ], + "notes": "found=['agents', 'claude', 'opencode'] leaked=[] tools_used=0 skill_loads=3 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".claude/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v1/print/S4/S4_all-roots/run1.json", + "out/opencode-v1/print/S4/S4_all-roots/run11.json" + ], + "notes": "found=['agents', 'claude', 'opencode'] leaked=[] tools_used=0 skill_loads=3 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".codex/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v1/print/S4/S4_all-roots/run13.json", + "out/opencode-v1/print/S4/S4_all-roots/run3.json" + ], + "notes": "found=['agents', 'claude', 'opencode'] leaked=[] tools_used=0 skill_loads=3 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v1/print/S4/S4_all-roots/run14.json", + "out/opencode-v1/print/S4/S4_all-roots/run4.json" + ], + "notes": "found=['agents', 'claude', 'opencode'] leaked=[] tools_used=0 skill_loads=3 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v1/print/S4/S4_all-roots/run16.json", + "out/opencode-v1/print/S4/S4_all-roots/run6.json" + ], + "notes": "found=['agents', 'claude', 'opencode'] leaked=[] tools_used=0 skill_loads=3 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".github/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v1/print/S4/S4_all-roots/run15.json", + "out/opencode-v1/print/S4/S4_all-roots/run5.json" + ], + "notes": "found=['agents', 'claude', 'opencode'] leaked=[] tools_used=0 skill_loads=3 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v1/print/S4/S4_all-roots/run17.json", + "out/opencode-v1/print/S4/S4_all-roots/run7.json" + ], + "notes": "found=['agents', 'claude', 'opencode'] leaked=[] tools_used=0 skill_loads=3 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v1/print/S4/S4_all-roots/run19.json", + "out/opencode-v1/print/S4/S4_all-roots/run9.json" + ], + "notes": "found=['agents', 'claude', 'opencode'] leaked=[] tools_used=0 skill_loads=3 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v1/print/S4/S4_all-roots/run18.json", + "out/opencode-v1/print/S4/S4_all-roots/run8.json" + ], + "notes": "found=['agents', 'claude', 'opencode'] leaked=[] tools_used=0 skill_loads=3 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-agent", + "root": ".agent/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/opencode-v1/print/S4/S4_confirm-agent/run1.json" + ], + "notes": "tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-codex", + "root": ".codex/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/opencode-v1/print/S4/S4_confirm-codex/run1.json" + ], + "notes": "tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-cursor", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/opencode-v1/print/S4/S4_confirm-cursor/run1.json" + ], + "notes": "tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-gemini", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/opencode-v1/print/S4/S4_confirm-gemini/run1.json" + ], + "notes": "tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-github", + "root": ".github/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/opencode-v1/print/S4/S4_confirm-github/run1.json" + ], + "notes": "tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-kiro", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/opencode-v1/print/S4/S4_confirm-kiro/run1.json" + ], + "notes": "tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-pi", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/opencode-v1/print/S4/S4_confirm-pi/run1.json" + ], + "notes": "tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S6", + "arm": "S6/delete", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "revoked", + "flaky": false, + "runs": 2, + "mechanism": "plugin event session.created", + "evidence": [ + "out/opencode-v1/print/S6/S6_delete/run1.json", + "out/opencode-v1/print/S6/S6_delete/run2.json" + ], + "notes": "tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S6", + "arm": "S6/update", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": "plugin event session.created", + "evidence": [ + "out/opencode-v1/print/S6/S6_update/run1.json", + "out/opencode-v1/print/S6/S6_update/run2.json" + ], + "notes": "tools_used=0 skill_loads=1 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S7", + "arm": "S7/add", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v1/print/S7/S7_add/run1.json", + "out/opencode-v1/print/S7/S7_add/run2.json" + ], + "notes": "turn1=not_visible session=ses_f5084adaeffeQCaEizJWHDHAiI tools_used=0 skill_loads=1 searches=0; turn1=not_visible session=ses_f50847c02ffedOoqojkkddscxF tools_used=0 skill_loads=1 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S7", + "arm": "S7/update", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v1/print/S7/S7_update/run1.json", + "out/opencode-v1/print/S7/S7_update/run2.json" + ], + "notes": "turn1=visible_first_turn session=ses_f508441d8ffeTykgwL8fiIvrk1 tools_used=0 skill_loads=1 searches=0; turn1=visible_first_turn session=ses_f5084049bffe9N8jCJK04BZnmM tools_used=0 skill_loads=1 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S8", + "arm": "S8/ancestor", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v1/print/S8/S8_ancestor/run1.json", + "out/opencode-v1/print/S8/S8_ancestor/run2.json" + ], + "notes": "cwd=sub/dir tools_used=0 skill_loads=1 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S8", + "arm": "S8/local", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v1/print/S8/S8_local/run1.json", + "out/opencode-v1/print/S8/S8_local/run2.json" + ], + "notes": "cwd=sub/dir tools_used=0 skill_loads=1 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S9", + "arm": "S9/linked-other", + "root": ".opencode/skills", + "exclusion": "info-exclude", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v1/print/S9/S9_linked-other/run1.json", + "out/opencode-v1/print/S9/S9_linked-other/run2.json" + ], + "notes": "cwd=wt-b skill_in=main tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S9", + "arm": "S9/linked-own", + "root": ".opencode/skills", + "exclusion": "info-exclude", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v1/print/S9/S9_linked-own/run1.json", + "out/opencode-v1/print/S9/S9_linked-own/run2.json" + ], + "notes": "cwd=wt-b skill_in=linked tools_used=0 skill_loads=1 searches=0" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S0", + "arm": "S0/none", + "root": null, + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v1/tui/S0/S0_none/run1.json", + "out/opencode-v1/tui/S0/S0_none/run2.json" + ], + "notes": "" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S1", + "arm": "S1/native", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v1/tui/S1/S1_native/run1.json", + "out/opencode-v1/tui/S1/S1_native/run2.json" + ], + "notes": "" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S2", + "arm": "S2/hook-adds-skill", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v1/tui/S2/S2_hook-adds-skill/run1.json", + "out/opencode-v1/tui/S2/S2_hook-adds-skill/run2.json" + ], + "notes": "exception=RuntimeError('no reply read from the screen; exit None; timeout no PROBE-REPLY line on the screen; stderr: 48;5;232magents\\x1b[0m\\x1b[30;124H\\x1b[38;5;255m\\x1b[48;5;232mctrl+p \\x1b[0m\\x1b[30;131H\\x1b[38;5;8m\\x1b[48;5;232mcommands\\x1b[0m\\x1b[49;3H\\x1b[38;5;8m\\x1b[48;5;232m/private/var/folders/lw/y5x_2wrn7w7g40gdhdkhv2m00000gn/T/skprobe-mgiit673/repo:main\\x1b[0m\\x1b[49;192H\\x1b[38;5;8m\\x1b[48;5;232m1.18.31\\x1b[0m\\x1b[0m\\x1b]12;#eeeeee\\x07\\x1b[1 q\\x1b[26;67H\\x1b[?25h\\x1b[?2026l'); exception=RuntimeError('no reply read from the screen; exit None; timeout no PROBE-REPLY line on the screen; stderr: 38;5;8m\\x1b[48;5;232mcommands\\x1b[0m\\x1b[49;3H\\x1b[38;5;8m\\x1b[48;5;232m/private/var/folders/lw/y5x_2wrn7w7g40gdhdkhv2m00000gn/T/skprobe-mpipbnqh/repo\\x1b[0m\\x1b[49;192H\\x1b[38;5;8m\\x1b[48;5;232m1.18.31\\x1b[0m\\x1b[0m\\x1b]12;#eeeeee\\x07\\x1b[1 q\\x1b[26;67H\\x1b[?25h\\x1b[?2026l\\x1b[?2026h\\x1b[?25l\\x1b[49;81H\\x1b[38;5;8m\\x1b[48;5;232m:main\\x1b[0m\\x1b[0m\\x1b[26;67H\\x1b[?25h\\x1b[?2026l')" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S5", + "arm": "S5/add", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v1/tui/S5/S5_add/run1.json", + "out/opencode-v1/tui/S5/S5_add/run2.json" + ], + "notes": "turn1=not_visible" + }, + { + "entry": "opencode-v1", + "harness": "opencode", + "version": "1.18.31", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S5", + "arm": "S5/reload", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/opencode-v1/tui/S5/S5_reload/run1.json" + ], + "notes": "no reload command for this entry" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S0", + "arm": "S0/none", + "root": null, + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v2/daemon/S0/S0_none/run1.json", + "out/opencode-v2/daemon/S0/S0_none/run2.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S1", + "arm": "S1/native", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v2/daemon/S1/S1_native/run1.json", + "out/opencode-v2/daemon/S1/S1_native/run2.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_loads=1 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S10", + "arm": "S10/hook-from-peer", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 2, + "mechanism": "plugin setup (no reload)", + "evidence": [ + "out/opencode-v2/daemon/S10/S10_hook-from-peer/run1.json", + "out/opencode-v2/daemon/S10/S10_hook-from-peer/run2.json" + ], + "notes": "turn1=not_visible peer=not_visible hook never fired stopReason=end_turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S2", + "arm": "S2/hook-adds-skill", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 2, + "mechanism": "plugin setup (no reload)", + "evidence": [ + "out/opencode-v2/daemon/S2/S2_hook-adds-skill/run1.json", + "out/opencode-v2/daemon/S2/S2_hook-adds-skill/run2.json" + ], + "notes": "hook never fired skill file absent after the turn stopReason=end_turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S2", + "arm": "S2/hook-creates-root", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 2, + "mechanism": "plugin setup (no reload)", + "evidence": [ + "out/opencode-v2/daemon/S2/S2_hook-creates-root/run1.json", + "out/opencode-v2/daemon/S2/S2_hook-creates-root/run2.json" + ], + "notes": "hook never fired skill file absent after the turn stopReason=end_turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S2", + "arm": "S2/registration", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 1, + "mechanism": "plugin setup + skill.reload", + "evidence": [ + "out/opencode-v2/daemon/S2/S2_registration/run1.json" + ], + "notes": "skill file absent after the turn stopReason=end_turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S3", + "arm": "S3/gitignore", + "root": ".opencode/skills", + "exclusion": "gitignore", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v2/daemon/S3/S3_gitignore/run1.json", + "out/opencode-v2/daemon/S3/S3_gitignore/run2.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_loads=1 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S3", + "arm": "S3/info-exclude", + "root": ".opencode/skills", + "exclusion": "info-exclude", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v2/daemon/S3/S3_info-exclude/run1.json", + "out/opencode-v2/daemon/S3/S3_info-exclude/run2.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_loads=1 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".agent/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 3, + "mechanism": null, + "evidence": [ + "out/opencode-v2/daemon/S4/S4_all-roots/run10.json", + "out/opencode-v2/daemon/S4/S4_all-roots/run20.json", + "out/opencode-v2/daemon/S4/S4_all-roots/run30.json" + ], + "notes": "found=[] leaked=[] stopReason=end_turn tools_used=0 skill_loads=0 searches=0; found=['agents', 'claude', 'opencode'] leaked=[] stopReason=end_turn tools_used=0 skill_loads=3 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": true, + "runs": 3, + "mechanism": null, + "evidence": [ + "out/opencode-v2/daemon/S4/S4_all-roots/run12.json", + "out/opencode-v2/daemon/S4/S4_all-roots/run2.json", + "out/opencode-v2/daemon/S4/S4_all-roots/run22.json" + ], + "notes": "found=['agents', 'claude', 'opencode'] leaked=[] stopReason=end_turn tools_used=0 skill_loads=3 searches=0; found=[] leaked=[] stopReason=end_turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".claude/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": true, + "runs": 3, + "mechanism": null, + "evidence": [ + "out/opencode-v2/daemon/S4/S4_all-roots/run1.json", + "out/opencode-v2/daemon/S4/S4_all-roots/run11.json", + "out/opencode-v2/daemon/S4/S4_all-roots/run21.json" + ], + "notes": "found=[] leaked=[] stopReason=end_turn tools_used=0 skill_loads=0 searches=0; found=['agents', 'claude', 'opencode'] leaked=[] stopReason=end_turn tools_used=0 skill_loads=3 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".codex/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 3, + "mechanism": null, + "evidence": [ + "out/opencode-v2/daemon/S4/S4_all-roots/run13.json", + "out/opencode-v2/daemon/S4/S4_all-roots/run23.json", + "out/opencode-v2/daemon/S4/S4_all-roots/run3.json" + ], + "notes": "found=['agents', 'claude', 'opencode'] leaked=[] stopReason=end_turn tools_used=0 skill_loads=3 searches=0; found=[] leaked=[] stopReason=end_turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 3, + "mechanism": null, + "evidence": [ + "out/opencode-v2/daemon/S4/S4_all-roots/run14.json", + "out/opencode-v2/daemon/S4/S4_all-roots/run24.json", + "out/opencode-v2/daemon/S4/S4_all-roots/run4.json" + ], + "notes": "found=['agents', 'claude', 'opencode'] leaked=[] stopReason=end_turn tools_used=0 skill_loads=3 searches=0; found=[] leaked=[] stopReason=end_turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 3, + "mechanism": null, + "evidence": [ + "out/opencode-v2/daemon/S4/S4_all-roots/run16.json", + "out/opencode-v2/daemon/S4/S4_all-roots/run26.json", + "out/opencode-v2/daemon/S4/S4_all-roots/run6.json" + ], + "notes": "found=['agents', 'claude', 'opencode'] leaked=[] stopReason=end_turn tools_used=0 skill_loads=3 searches=0; found=[] leaked=[] stopReason=end_turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".github/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 3, + "mechanism": null, + "evidence": [ + "out/opencode-v2/daemon/S4/S4_all-roots/run15.json", + "out/opencode-v2/daemon/S4/S4_all-roots/run25.json", + "out/opencode-v2/daemon/S4/S4_all-roots/run5.json" + ], + "notes": "found=['agents', 'claude', 'opencode'] leaked=[] stopReason=end_turn tools_used=0 skill_loads=3 searches=0; found=[] leaked=[] stopReason=end_turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 3, + "mechanism": null, + "evidence": [ + "out/opencode-v2/daemon/S4/S4_all-roots/run17.json", + "out/opencode-v2/daemon/S4/S4_all-roots/run27.json", + "out/opencode-v2/daemon/S4/S4_all-roots/run7.json" + ], + "notes": "found=['agents', 'claude', 'opencode'] leaked=[] stopReason=end_turn tools_used=0 skill_loads=3 searches=0; found=[] leaked=[] stopReason=end_turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": true, + "runs": 3, + "mechanism": null, + "evidence": [ + "out/opencode-v2/daemon/S4/S4_all-roots/run19.json", + "out/opencode-v2/daemon/S4/S4_all-roots/run29.json", + "out/opencode-v2/daemon/S4/S4_all-roots/run9.json" + ], + "notes": "found=['agents', 'claude', 'opencode'] leaked=[] stopReason=end_turn tools_used=0 skill_loads=3 searches=0; found=[] leaked=[] stopReason=end_turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 3, + "mechanism": null, + "evidence": [ + "out/opencode-v2/daemon/S4/S4_all-roots/run18.json", + "out/opencode-v2/daemon/S4/S4_all-roots/run28.json", + "out/opencode-v2/daemon/S4/S4_all-roots/run8.json" + ], + "notes": "found=['agents', 'claude', 'opencode'] leaked=[] stopReason=end_turn tools_used=0 skill_loads=3 searches=0; found=[] leaked=[] stopReason=end_turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-agent", + "root": ".agent/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/opencode-v2/daemon/S4/S4_confirm-agent/run1.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-agents", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/opencode-v2/daemon/S4/S4_confirm-agents/run1.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_loads=1 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-claude", + "root": ".claude/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/opencode-v2/daemon/S4/S4_confirm-claude/run1.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_loads=1 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-codex", + "root": ".codex/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/opencode-v2/daemon/S4/S4_confirm-codex/run1.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-cursor", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/opencode-v2/daemon/S4/S4_confirm-cursor/run1.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-gemini", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/opencode-v2/daemon/S4/S4_confirm-gemini/run1.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-github", + "root": ".github/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/opencode-v2/daemon/S4/S4_confirm-github/run1.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-kiro", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/opencode-v2/daemon/S4/S4_confirm-kiro/run1.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-opencode", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/opencode-v2/daemon/S4/S4_confirm-opencode/run1.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_loads=1 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-pi", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/opencode-v2/daemon/S4/S4_confirm-pi/run1.json" + ], + "notes": "stopReason=end_turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S5", + "arm": "S5/add", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": true, + "runs": 3, + "mechanism": null, + "evidence": [ + "out/opencode-v2/daemon/S5/S5_add/run1.json", + "out/opencode-v2/daemon/S5/S5_add/run2.json", + "out/opencode-v2/daemon/S5/S5_add/run3.json" + ], + "notes": "turn1=not_visible stopReason=end_turn tools_used=0 skill_loads=1 searches=0; turn1=not_visible stopReason=end_turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S5", + "arm": "S5/delete", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "revoked", + "flaky": true, + "runs": 3, + "mechanism": null, + "evidence": [ + "out/opencode-v2/daemon/S5/S5_delete/run1.json", + "out/opencode-v2/daemon/S5/S5_delete/run2.json", + "out/opencode-v2/daemon/S5/S5_delete/run3.json" + ], + "notes": "turn1=visible_first_turn stopReason=end_turn tools_used=0 skill_loads=1 searches=0; turn1=visible_first_turn stopReason=end_turn tools_used=0 skill_loads=0 searches=0; exception=RuntimeError('no reply read from the output; exit None; exception=TimeoutError() tools_used=0 skill_loads=0 searches=0; stderr: ')" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S5", + "arm": "S5/update", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "stale", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v2/daemon/S5/S5_update/run1.json", + "out/opencode-v2/daemon/S5/S5_update/run2.json" + ], + "notes": "turn1=visible_first_turn stopReason=end_turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S6", + "arm": "S6/delete", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 2, + "mechanism": "plugin setup (no reload)", + "evidence": [ + "out/opencode-v2/daemon/S6/S6_delete/run1.json", + "out/opencode-v2/daemon/S6/S6_delete/run2.json" + ], + "notes": "hook never fired stopReason=end_turn tools_used=0 skill_loads=1 searches=0; hook never fired stopReason=end_turn tools_used=2 skill_loads=0 searches=2" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S6", + "arm": "S6/update", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 2, + "mechanism": "plugin setup (no reload)", + "evidence": [ + "out/opencode-v2/daemon/S6/S6_update/run1.json", + "out/opencode-v2/daemon/S6/S6_update/run2.json" + ], + "notes": "hook never fired stopReason=end_turn tools_used=0 skill_loads=1 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S0", + "arm": "S0/none", + "root": null, + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v2/print/S0/S0_none/run1.json", + "out/opencode-v2/print/S0/S0_none/run2.json" + ], + "notes": "tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S1", + "arm": "S1/native", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v2/print/S1/S1_native/run1.json", + "out/opencode-v2/print/S1/S1_native/run2.json" + ], + "notes": "tools_used=0 skill_loads=1 searches=0; tools_used=7 skill_loads=1 searches=5" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S2", + "arm": "S2/hook-adds-skill", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 2, + "mechanism": "plugin setup (no reload)", + "evidence": [ + "out/opencode-v2/print/S2/S2_hook-adds-skill/run1.json", + "out/opencode-v2/print/S2/S2_hook-adds-skill/run2.json" + ], + "notes": "hook never fired skill file absent after the turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S2", + "arm": "S2/hook-creates-root", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 2, + "mechanism": "plugin setup (no reload)", + "evidence": [ + "out/opencode-v2/print/S2/S2_hook-creates-root/run1.json", + "out/opencode-v2/print/S2/S2_hook-creates-root/run2.json" + ], + "notes": "hook never fired skill file absent after the turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S2", + "arm": "S2/registration", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 1, + "mechanism": "plugin setup + skill.reload", + "evidence": [ + "out/opencode-v2/print/S2/S2_registration/run1.json" + ], + "notes": "skill file absent after the turn tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S3", + "arm": "S3/gitignore", + "root": ".opencode/skills", + "exclusion": "gitignore", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v2/print/S3/S3_gitignore/run1.json", + "out/opencode-v2/print/S3/S3_gitignore/run2.json" + ], + "notes": "tools_used=0 skill_loads=1 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S3", + "arm": "S3/info-exclude", + "root": ".opencode/skills", + "exclusion": "info-exclude", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v2/print/S3/S3_info-exclude/run1.json", + "out/opencode-v2/print/S3/S3_info-exclude/run2.json" + ], + "notes": "tools_used=0 skill_loads=1 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".agent/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 3, + "mechanism": null, + "evidence": [ + "out/opencode-v2/print/S4/S4_all-roots/run10.json", + "out/opencode-v2/print/S4/S4_all-roots/run20.json", + "out/opencode-v2/print/S4/S4_all-roots/run30.json" + ], + "notes": "found=[] leaked=[] tools_used=1 skill_loads=0 searches=0; found=['agents', 'claude', 'opencode'] leaked=[] tools_used=0 skill_loads=3 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": true, + "runs": 3, + "mechanism": null, + "evidence": [ + "out/opencode-v2/print/S4/S4_all-roots/run12.json", + "out/opencode-v2/print/S4/S4_all-roots/run2.json", + "out/opencode-v2/print/S4/S4_all-roots/run22.json" + ], + "notes": "found=['agents', 'claude', 'opencode'] leaked=[] tools_used=0 skill_loads=3 searches=0; found=[] leaked=[] tools_used=1 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".claude/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": true, + "runs": 3, + "mechanism": null, + "evidence": [ + "out/opencode-v2/print/S4/S4_all-roots/run1.json", + "out/opencode-v2/print/S4/S4_all-roots/run11.json", + "out/opencode-v2/print/S4/S4_all-roots/run21.json" + ], + "notes": "found=[] leaked=[] tools_used=1 skill_loads=0 searches=0; found=['agents', 'claude', 'opencode'] leaked=[] tools_used=0 skill_loads=3 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".codex/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 3, + "mechanism": null, + "evidence": [ + "out/opencode-v2/print/S4/S4_all-roots/run13.json", + "out/opencode-v2/print/S4/S4_all-roots/run23.json", + "out/opencode-v2/print/S4/S4_all-roots/run3.json" + ], + "notes": "found=['agents', 'claude', 'opencode'] leaked=[] tools_used=0 skill_loads=3 searches=0; found=[] leaked=[] tools_used=1 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 3, + "mechanism": null, + "evidence": [ + "out/opencode-v2/print/S4/S4_all-roots/run14.json", + "out/opencode-v2/print/S4/S4_all-roots/run24.json", + "out/opencode-v2/print/S4/S4_all-roots/run4.json" + ], + "notes": "found=['agents', 'claude', 'opencode'] leaked=[] tools_used=0 skill_loads=3 searches=0; found=[] leaked=[] tools_used=1 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 3, + "mechanism": null, + "evidence": [ + "out/opencode-v2/print/S4/S4_all-roots/run16.json", + "out/opencode-v2/print/S4/S4_all-roots/run26.json", + "out/opencode-v2/print/S4/S4_all-roots/run6.json" + ], + "notes": "found=['agents', 'claude', 'opencode'] leaked=[] tools_used=0 skill_loads=3 searches=0; found=[] leaked=[] tools_used=1 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".github/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 3, + "mechanism": null, + "evidence": [ + "out/opencode-v2/print/S4/S4_all-roots/run15.json", + "out/opencode-v2/print/S4/S4_all-roots/run25.json", + "out/opencode-v2/print/S4/S4_all-roots/run5.json" + ], + "notes": "found=['agents', 'claude', 'opencode'] leaked=[] tools_used=0 skill_loads=3 searches=0; found=[] leaked=[] tools_used=1 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 3, + "mechanism": null, + "evidence": [ + "out/opencode-v2/print/S4/S4_all-roots/run17.json", + "out/opencode-v2/print/S4/S4_all-roots/run27.json", + "out/opencode-v2/print/S4/S4_all-roots/run7.json" + ], + "notes": "found=['agents', 'claude', 'opencode'] leaked=[] tools_used=0 skill_loads=3 searches=0; found=[] leaked=[] tools_used=1 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": true, + "runs": 3, + "mechanism": null, + "evidence": [ + "out/opencode-v2/print/S4/S4_all-roots/run19.json", + "out/opencode-v2/print/S4/S4_all-roots/run29.json", + "out/opencode-v2/print/S4/S4_all-roots/run9.json" + ], + "notes": "found=['agents', 'claude', 'opencode'] leaked=[] tools_used=0 skill_loads=3 searches=0; found=[] leaked=[] tools_used=1 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 3, + "mechanism": null, + "evidence": [ + "out/opencode-v2/print/S4/S4_all-roots/run18.json", + "out/opencode-v2/print/S4/S4_all-roots/run28.json", + "out/opencode-v2/print/S4/S4_all-roots/run8.json" + ], + "notes": "found=['agents', 'claude', 'opencode'] leaked=[] tools_used=0 skill_loads=3 searches=0; found=[] leaked=[] tools_used=1 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-agent", + "root": ".agent/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/opencode-v2/print/S4/S4_confirm-agent/run1.json" + ], + "notes": "tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-agents", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/opencode-v2/print/S4/S4_confirm-agents/run1.json" + ], + "notes": "tools_used=0 skill_loads=1 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-claude", + "root": ".claude/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/opencode-v2/print/S4/S4_confirm-claude/run1.json" + ], + "notes": "tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-codex", + "root": ".codex/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/opencode-v2/print/S4/S4_confirm-codex/run1.json" + ], + "notes": "tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-cursor", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/opencode-v2/print/S4/S4_confirm-cursor/run1.json" + ], + "notes": "tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-gemini", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/opencode-v2/print/S4/S4_confirm-gemini/run1.json" + ], + "notes": "tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-github", + "root": ".github/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/opencode-v2/print/S4/S4_confirm-github/run1.json" + ], + "notes": "tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-kiro", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/opencode-v2/print/S4/S4_confirm-kiro/run1.json" + ], + "notes": "tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-opencode", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/opencode-v2/print/S4/S4_confirm-opencode/run1.json" + ], + "notes": "tools_used=0 skill_loads=1 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-pi", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/opencode-v2/print/S4/S4_confirm-pi/run1.json" + ], + "notes": "tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S6", + "arm": "S6/delete", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 2, + "mechanism": "plugin setup (no reload)", + "evidence": [ + "out/opencode-v2/print/S6/S6_delete/run1.json", + "out/opencode-v2/print/S6/S6_delete/run2.json" + ], + "notes": "hook never fired tools_used=3 skill_loads=0 searches=3; hook never fired tools_used=0 skill_loads=1 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S6", + "arm": "S6/update", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 2, + "mechanism": "plugin setup (no reload)", + "evidence": [ + "out/opencode-v2/print/S6/S6_update/run1.json", + "out/opencode-v2/print/S6/S6_update/run2.json" + ], + "notes": "hook never fired tools_used=0 skill_loads=1 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S7", + "arm": "S7/add", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v2/print/S7/S7_add/run1.json", + "out/opencode-v2/print/S7/S7_add/run2.json" + ], + "notes": "turn1=not_visible session=ses_f507cb360ffeGnQk8CQ86lQRyz tools_used=0 skill_loads=1 searches=0; turn1=not_visible session=ses_f507c8986ffe7X6NxX2CYFC46z tools_used=0 skill_loads=1 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S7", + "arm": "S7/update", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v2/print/S7/S7_update/run1.json", + "out/opencode-v2/print/S7/S7_update/run2.json" + ], + "notes": "turn1=visible_first_turn session=ses_f507bffdeffeMTm5300ywFQWEZ tools_used=0 skill_loads=1 searches=0; turn1=visible_first_turn session=ses_f507bc320ffe8AWRORDpgG06VI tools_used=0 skill_loads=1 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S8", + "arm": "S8/ancestor", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v2/print/S8/S8_ancestor/run1.json", + "out/opencode-v2/print/S8/S8_ancestor/run2.json" + ], + "notes": "cwd=sub/dir tools_used=0 skill_loads=1 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S8", + "arm": "S8/local", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v2/print/S8/S8_local/run1.json", + "out/opencode-v2/print/S8/S8_local/run2.json" + ], + "notes": "cwd=sub/dir tools_used=1 skill_loads=1 searches=1; cwd=sub/dir tools_used=0 skill_loads=1 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S9", + "arm": "S9/linked-other", + "root": ".opencode/skills", + "exclusion": "info-exclude", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v2/print/S9/S9_linked-other/run1.json", + "out/opencode-v2/print/S9/S9_linked-other/run2.json" + ], + "notes": "cwd=wt-b skill_in=main tools_used=0 skill_loads=0 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S9", + "arm": "S9/linked-own", + "root": ".opencode/skills", + "exclusion": "info-exclude", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v2/print/S9/S9_linked-own/run1.json", + "out/opencode-v2/print/S9/S9_linked-own/run2.json" + ], + "notes": "cwd=wt-b skill_in=linked tools_used=0 skill_loads=1 searches=0" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S0", + "arm": "S0/none", + "root": null, + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v2/tui/S0/S0_none/run1.json", + "out/opencode-v2/tui/S0/S0_none/run2.json" + ], + "notes": "" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S1", + "arm": "S1/native", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v2/tui/S1/S1_native/run1.json", + "out/opencode-v2/tui/S1/S1_native/run2.json" + ], + "notes": "" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S2", + "arm": "S2/hook-adds-skill", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 2, + "mechanism": "plugin setup (no reload)", + "evidence": [ + "out/opencode-v2/tui/S2/S2_hook-adds-skill/run1.json", + "out/opencode-v2/tui/S2/S2_hook-adds-skill/run2.json" + ], + "notes": "hook never fired skill file absent after the turn" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S5", + "arm": "S5/add", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "visible_live", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/opencode-v2/tui/S5/S5_add/run1.json", + "out/opencode-v2/tui/S5/S5_add/run2.json" + ], + "notes": "turn1=not_visible" + }, + { + "entry": "opencode-v2", + "harness": "opencode", + "version": "opencode v2.0.4", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S5", + "arm": "S5/reload", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/opencode-v2/tui/S5/S5_reload/run1.json" + ], + "notes": "no reload command for this entry" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S0", + "arm": "S0/none", + "root": null, + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/pi/daemon/S0/S0_none/run1.json", + "out/pi/daemon/S0/S0_none/run2.json" + ], + "notes": "tools_used=0; tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S1", + "arm": "S1/native", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/pi/daemon/S1/S1_native/run1.json", + "out/pi/daemon/S1/S1_native/run2.json" + ], + "notes": "tools_used=1; tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S10", + "arm": "S10/hook-from-peer", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": "extension session_start", + "evidence": [ + "out/pi/daemon/S10/S10_hook-from-peer/run1.json", + "out/pi/daemon/S10/S10_hook-from-peer/run2.json" + ], + "notes": "turn1=not_visible peer=not_visible tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S2", + "arm": "S2/hook-adds-skill", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": "extension session_start", + "evidence": [ + "out/pi/daemon/S2/S2_hook-adds-skill/run1.json", + "out/pi/daemon/S2/S2_hook-adds-skill/run2.json" + ], + "notes": "tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S2", + "arm": "S2/hook-creates-root", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": "extension session_start", + "evidence": [ + "out/pi/daemon/S2/S2_hook-creates-root/run1.json", + "out/pi/daemon/S2/S2_hook-creates-root/run2.json" + ], + "notes": "tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S2", + "arm": "S2/registration", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "visible_after_reload", + "flaky": false, + "runs": 2, + "mechanism": "extension session_start + resources_discover skillPaths", + "evidence": [ + "out/pi/daemon/S2/S2_registration/run1.json", + "out/pi/daemon/S2/S2_registration/run2.json" + ], + "notes": "tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S3", + "arm": "S3/gitignore", + "root": ".pi/skills", + "exclusion": "gitignore", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/pi/daemon/S3/S3_gitignore/run1.json", + "out/pi/daemon/S3/S3_gitignore/run2.json" + ], + "notes": "tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S3", + "arm": "S3/info-exclude", + "root": ".pi/skills", + "exclusion": "info-exclude", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/pi/daemon/S3/S3_info-exclude/run1.json", + "out/pi/daemon/S3/S3_info-exclude/run2.json" + ], + "notes": "tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".agent/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/pi/daemon/S4/S4_all-roots/run10.json", + "out/pi/daemon/S4/S4_all-roots/run20.json" + ], + "notes": "found=['agents', 'pi'] leaked=[] tools_used=0 skill_reads=2 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/pi/daemon/S4/S4_all-roots/run12.json", + "out/pi/daemon/S4/S4_all-roots/run2.json" + ], + "notes": "found=['agents', 'pi'] leaked=[] tools_used=0 skill_reads=2 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".claude/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/pi/daemon/S4/S4_all-roots/run1.json", + "out/pi/daemon/S4/S4_all-roots/run11.json" + ], + "notes": "found=['agents', 'pi'] leaked=[] tools_used=0 skill_reads=2 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".codex/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/pi/daemon/S4/S4_all-roots/run13.json", + "out/pi/daemon/S4/S4_all-roots/run3.json" + ], + "notes": "found=['agents', 'pi'] leaked=[] tools_used=0 skill_reads=2 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/pi/daemon/S4/S4_all-roots/run14.json", + "out/pi/daemon/S4/S4_all-roots/run4.json" + ], + "notes": "found=['agents', 'pi'] leaked=[] tools_used=0 skill_reads=2 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/pi/daemon/S4/S4_all-roots/run16.json", + "out/pi/daemon/S4/S4_all-roots/run6.json" + ], + "notes": "found=['agents', 'pi'] leaked=[] tools_used=0 skill_reads=2 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".github/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/pi/daemon/S4/S4_all-roots/run15.json", + "out/pi/daemon/S4/S4_all-roots/run5.json" + ], + "notes": "found=['agents', 'pi'] leaked=[] tools_used=0 skill_reads=2 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/pi/daemon/S4/S4_all-roots/run17.json", + "out/pi/daemon/S4/S4_all-roots/run7.json" + ], + "notes": "found=['agents', 'pi'] leaked=[] tools_used=0 skill_reads=2 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/pi/daemon/S4/S4_all-roots/run19.json", + "out/pi/daemon/S4/S4_all-roots/run9.json" + ], + "notes": "found=['agents', 'pi'] leaked=[] tools_used=0 skill_reads=2 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/pi/daemon/S4/S4_all-roots/run18.json", + "out/pi/daemon/S4/S4_all-roots/run8.json" + ], + "notes": "found=['agents', 'pi'] leaked=[] tools_used=0 skill_reads=2 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-agent", + "root": ".agent/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/pi/daemon/S4/S4_confirm-agent/run1.json" + ], + "notes": "tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-claude", + "root": ".claude/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/pi/daemon/S4/S4_confirm-claude/run1.json" + ], + "notes": "tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-codex", + "root": ".codex/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/pi/daemon/S4/S4_confirm-codex/run1.json" + ], + "notes": "tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-cursor", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/pi/daemon/S4/S4_confirm-cursor/run1.json" + ], + "notes": "tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-gemini", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/pi/daemon/S4/S4_confirm-gemini/run1.json" + ], + "notes": "tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-github", + "root": ".github/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/pi/daemon/S4/S4_confirm-github/run1.json" + ], + "notes": "tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-kiro", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/pi/daemon/S4/S4_confirm-kiro/run1.json" + ], + "notes": "tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S4", + "arm": "S4/confirm-opencode", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/pi/daemon/S4/S4_confirm-opencode/run1.json" + ], + "notes": "tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S5", + "arm": "S5/add", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/pi/daemon/S5/S5_add/run1.json", + "out/pi/daemon/S5/S5_add/run2.json" + ], + "notes": "turn1=not_visible tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S5", + "arm": "S5/delete", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/pi/daemon/S5/S5_delete/run1.json", + "out/pi/daemon/S5/S5_delete/run2.json" + ], + "notes": "turn1=visible_first_turn tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S5", + "arm": "S5/update", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "visible_live", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/pi/daemon/S5/S5_update/run1.json", + "out/pi/daemon/S5/S5_update/run2.json" + ], + "notes": "turn1=visible_first_turn tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S6", + "arm": "S6/delete", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "revoked", + "flaky": false, + "runs": 2, + "mechanism": "extension session_start", + "evidence": [ + "out/pi/daemon/S6/S6_delete/run1.json", + "out/pi/daemon/S6/S6_delete/run2.json" + ], + "notes": "tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "daemon", + "scenario": "S6", + "arm": "S6/update", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": "extension session_start", + "evidence": [ + "out/pi/daemon/S6/S6_update/run1.json", + "out/pi/daemon/S6/S6_update/run2.json" + ], + "notes": "tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S0", + "arm": "S0/none", + "root": null, + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/pi/print/S0/S0_none/run1.json", + "out/pi/print/S0/S0_none/run2.json" + ], + "notes": "tools_used=0; tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S1", + "arm": "S1/native", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/pi/print/S1/S1_native/run1.json", + "out/pi/print/S1/S1_native/run2.json" + ], + "notes": "tools_used=1; tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S2", + "arm": "S2/hook-adds-skill", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": "extension session_start", + "evidence": [ + "out/pi/print/S2/S2_hook-adds-skill/run1.json", + "out/pi/print/S2/S2_hook-adds-skill/run2.json" + ], + "notes": "tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S2", + "arm": "S2/hook-creates-root", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": "extension session_start", + "evidence": [ + "out/pi/print/S2/S2_hook-creates-root/run1.json", + "out/pi/print/S2/S2_hook-creates-root/run2.json" + ], + "notes": "tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S2", + "arm": "S2/registration", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "visible_after_reload", + "flaky": false, + "runs": 2, + "mechanism": "extension session_start + resources_discover skillPaths", + "evidence": [ + "out/pi/print/S2/S2_registration/run1.json", + "out/pi/print/S2/S2_registration/run2.json" + ], + "notes": "tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S3", + "arm": "S3/gitignore", + "root": ".pi/skills", + "exclusion": "gitignore", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/pi/print/S3/S3_gitignore/run1.json", + "out/pi/print/S3/S3_gitignore/run2.json" + ], + "notes": "tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S3", + "arm": "S3/info-exclude", + "root": ".pi/skills", + "exclusion": "info-exclude", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/pi/print/S3/S3_info-exclude/run1.json", + "out/pi/print/S3/S3_info-exclude/run2.json" + ], + "notes": "tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".agent/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/pi/print/S4/S4_all-roots/run10.json", + "out/pi/print/S4/S4_all-roots/run20.json" + ], + "notes": "found=['agents', 'pi'] leaked=[] tools_used=0 skill_reads=2 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".agents/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/pi/print/S4/S4_all-roots/run12.json", + "out/pi/print/S4/S4_all-roots/run2.json" + ], + "notes": "found=['agents', 'pi'] leaked=[] tools_used=0 skill_reads=2 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".claude/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/pi/print/S4/S4_all-roots/run1.json", + "out/pi/print/S4/S4_all-roots/run11.json" + ], + "notes": "found=['agents', 'pi'] leaked=[] tools_used=0 skill_reads=2 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".codex/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/pi/print/S4/S4_all-roots/run13.json", + "out/pi/print/S4/S4_all-roots/run3.json" + ], + "notes": "found=['agents', 'pi'] leaked=[] tools_used=0 skill_reads=2 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/pi/print/S4/S4_all-roots/run14.json", + "out/pi/print/S4/S4_all-roots/run4.json" + ], + "notes": "found=['agents', 'pi'] leaked=[] tools_used=0 skill_reads=2 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/pi/print/S4/S4_all-roots/run16.json", + "out/pi/print/S4/S4_all-roots/run6.json" + ], + "notes": "found=['agents', 'pi'] leaked=[] tools_used=0 skill_reads=2 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".github/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/pi/print/S4/S4_all-roots/run15.json", + "out/pi/print/S4/S4_all-roots/run5.json" + ], + "notes": "found=['agents', 'pi'] leaked=[] tools_used=0 skill_reads=2 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/pi/print/S4/S4_all-roots/run17.json", + "out/pi/print/S4/S4_all-roots/run7.json" + ], + "notes": "found=['agents', 'pi'] leaked=[] tools_used=0 skill_reads=2 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/pi/print/S4/S4_all-roots/run19.json", + "out/pi/print/S4/S4_all-roots/run9.json" + ], + "notes": "found=['agents', 'pi'] leaked=[] tools_used=0 skill_reads=2 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/all-roots", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/pi/print/S4/S4_all-roots/run18.json", + "out/pi/print/S4/S4_all-roots/run8.json" + ], + "notes": "found=['agents', 'pi'] leaked=[] tools_used=0 skill_reads=2 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-agent", + "root": ".agent/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/pi/print/S4/S4_confirm-agent/run1.json" + ], + "notes": "tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-claude", + "root": ".claude/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/pi/print/S4/S4_confirm-claude/run1.json" + ], + "notes": "tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-codex", + "root": ".codex/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/pi/print/S4/S4_confirm-codex/run1.json" + ], + "notes": "tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-cursor", + "root": ".cursor/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/pi/print/S4/S4_confirm-cursor/run1.json" + ], + "notes": "tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-gemini", + "root": ".gemini/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/pi/print/S4/S4_confirm-gemini/run1.json" + ], + "notes": "tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-github", + "root": ".github/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/pi/print/S4/S4_confirm-github/run1.json" + ], + "notes": "tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-kiro", + "root": ".kiro/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/pi/print/S4/S4_confirm-kiro/run1.json" + ], + "notes": "tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S4", + "arm": "S4/confirm-opencode", + "root": ".opencode/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/pi/print/S4/S4_confirm-opencode/run1.json" + ], + "notes": "tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S6", + "arm": "S6/delete", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "revoked", + "flaky": false, + "runs": 2, + "mechanism": "extension session_start", + "evidence": [ + "out/pi/print/S6/S6_delete/run1.json", + "out/pi/print/S6/S6_delete/run2.json" + ], + "notes": "tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S6", + "arm": "S6/update", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": "extension session_start", + "evidence": [ + "out/pi/print/S6/S6_update/run1.json", + "out/pi/print/S6/S6_update/run2.json" + ], + "notes": "tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S7", + "arm": "S7/add", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/pi/print/S7/S7_add/run1.json", + "out/pi/print/S7/S7_add/run2.json" + ], + "notes": "turn1=not_visible session=01a0af6d-517f-7499-96c8-f0adcd0874dd tools_used=0 skill_reads=1 searches=0; turn1=not_visible session=01a0af6d-d8a4-70ed-abaf-252aca5b7017 tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S7", + "arm": "S7/update", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/pi/print/S7/S7_update/run1.json", + "out/pi/print/S7/S7_update/run2.json" + ], + "notes": "turn1=visible_first_turn session=01a0af6e-4519-71cf-998c-bab6b4bd9a06 tools_used=0 skill_reads=1 searches=0; turn1=visible_first_turn session=01a0af6e-b89a-7635-afd8-5d04f52a2b4d tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S8", + "arm": "S8/ancestor", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/pi/print/S8/S8_ancestor/run1.json", + "out/pi/print/S8/S8_ancestor/run2.json" + ], + "notes": "cwd=sub/dir tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S8", + "arm": "S8/local", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/pi/print/S8/S8_local/run1.json", + "out/pi/print/S8/S8_local/run2.json" + ], + "notes": "cwd=sub/dir tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S9", + "arm": "S9/linked-other", + "root": ".pi/skills", + "exclusion": "info-exclude", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/pi/print/S9/S9_linked-other/run1.json", + "out/pi/print/S9/S9_linked-other/run2.json" + ], + "notes": "cwd=wt-b skill_in=main tools_used=0 skill_reads=0 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "print", + "scenario": "S9", + "arm": "S9/linked-own", + "root": ".pi/skills", + "exclusion": "info-exclude", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/pi/print/S9/S9_linked-own/run1.json", + "out/pi/print/S9/S9_linked-own/run2.json" + ], + "notes": "cwd=wt-b skill_in=linked tools_used=0 skill_reads=1 searches=0" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S0", + "arm": "S0/none", + "root": null, + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/pi/tui/S0/S0_none/run1.json", + "out/pi/tui/S0/S0_none/run2.json" + ], + "notes": "" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S1", + "arm": "S1/native", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "visible_first_turn", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/pi/tui/S1/S1_native/run1.json", + "out/pi/tui/S1/S1_native/run2.json" + ], + "notes": "" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S2", + "arm": "S2/hook-adds-skill", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": "extension session_start", + "evidence": [ + "out/pi/tui/S2/S2_hook-adds-skill/run1.json", + "out/pi/tui/S2/S2_hook-adds-skill/run2.json" + ], + "notes": "" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S5", + "arm": "S5/add", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "not_visible", + "flaky": false, + "runs": 2, + "mechanism": null, + "evidence": [ + "out/pi/tui/S5/S5_add/run1.json", + "out/pi/tui/S5/S5_add/run2.json" + ], + "notes": "turn1=not_visible" + }, + { + "entry": "pi", + "harness": "pi", + "version": "0.85.1", + "os": "macOS 26.6.2 arm64", + "mode": "tui", + "scenario": "S5", + "arm": "S5/reload", + "root": ".pi/skills", + "exclusion": "none", + "verdict": "untested", + "flaky": false, + "runs": 1, + "mechanism": null, + "evidence": [ + "out/pi/tui/S5/S5_reload/run1.json" + ], + "notes": "no reload command for this entry" + } +] diff --git a/docs/probes/2026-09-16-skills-discovery/probe.py b/docs/probes/2026-09-16-skills-discovery/probe.py new file mode 100644 index 0000000000..4abf8cfea5 --- /dev/null +++ b/docs/probes/2026-09-16-skills-discovery/probe.py @@ -0,0 +1,816 @@ +#!/usr/bin/env python3 +"""Skills discovery probes: does a repo-local skill written at startup reach the first model request? + +Usage: + probe.py [--harness ENTRY ...] [--mode print|daemon|tui] [--scenario S0..S10 ...] [--turn] + [--runs N] [--outdir DIR] [--keep] [--rerun] [--emit] [--matrix FILE] [--base DIR] + +Without --turn only the free phase runs (binary, version, auth in an isolated root): zero model +requests. Each turn arm costs one model request per run. + +An arm whose output directory already holds its runs is not run again: a sweep interrupted +halfway resumes where it stopped instead of re-spending the turns it already paid for. An arm +with fewer runs than asked for is topped up, and one carrying an `untested` row is measured +again, so a missing binary or a broken control never latches. --rerun deletes an arm's +directory before running it, which is how a settled arm is re-measured. +""" +from __future__ import annotations + +import argparse +import json +import shutil +import sys +import time +from pathlib import Path +from typing import Callable + +KIT = Path(__file__).resolve().parent +sys.path.insert(0, str(KIT)) + +from harness import ENTRIES # noqa: E402 +from harness.base import Adapter, AskResult, Session # noqa: E402 +from lib.git_exclusion import apply as apply_exclusion, assert_untracked_state # noqa: E402 +from lib.hook_script import read_stamp, stamp_path, write_hook_script # noqa: E402 +from lib.isolation import Sandbox, add_worktree, new_sandbox # noqa: E402 +from lib.probe_skill import ( # noqa: E402 + ProbeSkill, multi_prompt, parse_reply, single_prompt, tui_prompt, write_skill, +) +from lib.recorder import ( # noqa: E402 + RunRecord, emit_matrix, load_runs, next_run_path, os_label, run_dir, write_run, +) +from lib.verdict import ( # noqa: E402 + PromptDesignFailure, judge_control, judge_delete, judge_root, judge_single, judge_update, + needs_third_run, combine, +) + +ALL_ROOTS: dict[str, str] = { + "claude": ".claude/skills", "agents": ".agents/skills", "codex": ".codex/skills", + "cursor": ".cursor/skills", "github": ".github/skills", "gemini": ".gemini/skills", + "kiro": ".kiro/skills", "pi": ".pi/skills", "opencode": ".opencode/skills", "agent": ".agent/skills", +} +SCENARIOS = ("S0", "S1", "S2", "S3", "S4", "S5", "S6", "S7", "S8", "S9", "S10") +S2_ARMS = ("hook-creates-root", "hook-adds-skill", "registration") +S3_ARMS = ("gitignore", "info-exclude") +S5_ARMS = ("add", "update", "delete") +S5_TUI_ARMS = ("add", "reload") +S6_ARMS = ("update", "delete") +S7_ARMS = ("add", "update") +S8_ARMS = ("ancestor", "local") +S9_ARMS = ("linked-own", "linked-other") +# Discovery does not depend on the launch mode, so the discovery-only scenarios run once, in the +# cheapest mode; the lifecycle scenarios run where a second turn exists. +MODE_SCENARIOS: dict[str, tuple[str, ...]] = { + "print": ("S0", "S1", "S2", "S3", "S4", "S6", "S7", "S8", "S9"), + "daemon": ("S0", "S1", "S2", "S3", "S4", "S5", "S6", "S10"), + "tui": ("S0", "S1", "S2", "S5"), +} + + +# The arms of a scenario, per mode: a sweep and the blocked rows it stands in for read the same +# table, so a mode cannot record a row for an arm it never runs. +SCENARIO_ARMS: dict[str, tuple[str, ...]] = { + "S0": ("none",), "S1": ("native",), "S2": S2_ARMS, "S3": S3_ARMS, "S5": S5_ARMS, "S6": S6_ARMS, + "S7": S7_ARMS, "S8": S8_ARMS, "S9": S9_ARMS, "S10": ("hook-from-peer",), +} +TUI_SCENARIO_ARMS: dict[str, tuple[str, ...]] = {"S2": ("hook-adds-skill",), "S5": S5_TUI_ARMS} + + +def arms_of(mode: str, scenario: str) -> tuple[str, ...]: + if mode == "tui" and scenario in TUI_SCENARIO_ARMS: + return TUI_SCENARIO_ARMS[scenario] + return SCENARIO_ARMS[scenario] + + +def arms_for(mode: str) -> list[tuple[str, str, bool, str]]: + """Every arm a full sweep runs in this mode, so an entry that cannot run gets a row per arm.""" + wanted = MODE_SCENARIOS[mode] + out: list[tuple[str, str, bool, str]] = [("S0", "S0/none", False, "none"), ("S1", "S1/native", True, "none")] + out += [("S2", f"S2/{a}", True, "none") for a in arms_of(mode, "S2")] + if "S3" in wanted: + out += [("S3", f"S3/{e}", True, e) for e in S3_ARMS] + if "S4" in wanted: + out.append(("S4", "S4/all-roots", False, "none")) + if "S5" in wanted: + out += [("S5", f"S5/{a}", True, "none") for a in arms_of(mode, "S5")] + if "S6" in wanted: + out += [("S6", f"S6/{a}", True, "none") for a in S6_ARMS] + if "S7" in wanted: + out += [("S7", f"S7/{a}", True, "none") for a in S7_ARMS] + if "S8" in wanted: + out += [("S8", f"S8/{a}", True, "none") for a in S8_ARMS] + if "S9" in wanted: + out += [("S9", f"S9/{a}", True, "info-exclude") for a in S9_ARMS] + if "S10" in wanted: + out.append(("S10", "S10/hook-from-peer", True, "none")) + return out + + +def _verdicts_by_root(recs: list[RunRecord]) -> dict[str | None, str]: + return {r.root: r.verdict for r in recs} + + +class Runner: + def __init__(self, adapter: Adapter, outdir: Path, runs: int = 2, keep: bool = False, + base: Path | None = None, version: str | None = None, rerun: bool = False) -> None: + self.adapter = adapter + self.outdir = outdir + self.runs = runs + self.keep = keep + self.base = base + self.rerun = rerun + self._cleared: set[Path] = set() + self._sb: Sandbox | None = None + self.s1_ok: dict[str, bool] = {} + self._version = version if version is not None else adapter.version() + self._binary = adapter.binary_path() or adapter.binary + self._os = os_label() + + # -- sandbox plumbing --------------------------------------------------------------------- + + def sandbox(self) -> Sandbox: + """The arm's sandbox; the guard that ran the arm tears it down, so a throwing arm still + gets its stderr copied before the directory goes.""" + a = self.adapter + sb = new_sandbox(a.lever, a.real_root(), list(a.credential_files), list(a.passthrough_env), + dict(a.extra_env), keep=self.keep, base=self.base) + try: + a.prepare(sb) + except Exception: + sb.cleanup() + raise + self._sb = sb + return sb + + def _teardown(self) -> None: + if self._sb is not None: + self._sb.cleanup() + self._sb = None + + def _s4_roots(self) -> dict[str, str]: + a = self.adapter + roots = dict(ALL_ROOTS) + if a.native_root not in roots.values(): + roots[f"native_{a.entry}"] = a.native_root + return roots + + def record(self, mode: str, scenario: str, arm: str, root: str | None, exclusion: str, + res: AskResult | None, verdict: str, expected: dict[str, str], hook: dict | None = None, + sb: Sandbox | None = None, notes: str = "", started: float | None = None, + auth_ok: bool | None = None, name: str | None = None, + prior: AskResult | None = None) -> RunRecord: + reply = parse_reply(res.reply_text, self._raw_for(mode, res), name) if res else None + rec = RunRecord( + entry=self.adapter.entry, harness=self.adapter.harness, binary=self._binary, + version=self._version, os=self._os, mode=mode, argv=res.argv if res else [], + isolation_lever=self.adapter.lever, credential_files=list(self.adapter.credential_files), + auth_ok=auth_ok, scenario=scenario, arm=arm, root=root, exclusion=exclusion, hook=hook, + first_request_at=res.first_request_at if res else None, reply=res.reply_text if res else "", + tokens_found=sorted(reply.tokens) if reply else [], skill_named=reply.skill_named if reply else False, + stderr_path=res.stderr_path if res else None, verdict=verdict, + duration_ms=int((time.time() - (started or time.time())) * 1000), + expected_tokens=expected, notes=(notes + ("" if not res or not res.notes else f" {res.notes}")).strip(), + ) + run_path = next_run_path(self.outdir, rec) + if sb is not None: + rec.stderr_path = self._keep_stderr(sb, run_path, rec.stderr_path) + if res is not None and res.raw: + # The vendor's full event stream is what a doubtful verdict is re-read against. + run_path.parent.mkdir(parents=True, exist_ok=True) + (run_path.parent / f"{run_path.stem}.raw.txt").write_text(res.raw) + if prior is not None and prior.raw: + run_path.parent.mkdir(parents=True, exist_ok=True) + (run_path.parent / f"{run_path.stem}.prior.raw.txt").write_text(prior.raw) + write_run(self.outdir, rec) + return rec + + def _keep_stderr(self, sb: Sandbox, run_path: Path, stderr_path: str | None) -> str | None: + """Copy the sandbox's stderr logs beside the run file: the sandbox is about to be removed.""" + wanted = {str(p) for p in sb.root.rglob("*.stderr.log")} | {str(p) for p in sb.root.rglob("*-tui.log")} + if stderr_path and Path(stderr_path).is_file() and Path(stderr_path).is_relative_to(sb.root): + wanted.add(stderr_path) + for src in sorted(wanted): + if src not in sb.copied_logs: + dst = run_path.parent / f"{run_path.stem}.{Path(src).name}" + shutil.copy2(src, dst) + sb.copied_logs[src] = str(dst) + if stderr_path in sb.copied_logs: + return sb.copied_logs[stderr_path] + if stderr_path is None and len(sb.copied_logs) == 1: + return next(iter(sb.copied_logs.values())) + return stderr_path + + def _load(self, mode: str, scenario: str, arm: str) -> tuple[Path, list[RunRecord]]: + d = run_dir(self.outdir, self.adapter.entry, mode, scenario, arm) + if self.rerun and d not in self._cleared: + self._cleared.add(d) + shutil.rmtree(d, ignore_errors=True) + recs = load_runs(d) if (d / "run1.json").exists() else [] + recs.sort(key=lambda r: int(r._path.stem[3:])) # type: ignore[attr-defined] + return d, recs + + def _existing(self, mode: str, scenario: str, arm: str, per_pass: int = 1) -> list[RunRecord]: + """The arm's settled runs, or [] after clearing rows nothing can resume from: an untested + row is a failure to measure, not a measurement, and a pass cut short must be redone.""" + d, recs = self._load(mode, scenario, arm) + if not recs: + return [] + if any(r.verdict == "untested" for r in recs) or len(recs) % per_pass: + shutil.rmtree(d, ignore_errors=True) + return [] + return recs + + def _guarded(self, fn: Callable[[], RunRecord | list[RunRecord]], mode: str, scenario: str, + arm: str, root: str | None, exclusion: str) -> list[RunRecord]: + try: + out = fn() + except PromptDesignFailure: + raise + except Exception as ex: # noqa: BLE001 + return [self.record(mode, scenario, arm, root, exclusion, None, "untested", {}, sb=self._sb, + notes=f"exception={ex!r}")] + finally: + self._teardown() + return out if isinstance(out, list) else [out] + + def run_arm(self, fn: Callable[[], RunRecord], mode: str, scenario: str, arm: str, + root: str | None, exclusion: str) -> list[RunRecord]: + recs = list(self._existing(mode, scenario, arm)) + for _ in range(max(0, self.runs - len(recs))): + recs += self._guarded(fn, mode, scenario, arm, root, exclusion) + if needs_third_run([r.verdict for r in recs]): + recs += self._guarded(fn, mode, scenario, arm, root, exclusion) + return recs + + def run_once(self, fn: Callable[[], RunRecord], mode: str, scenario: str, arm: str, + root: str | None, exclusion: str) -> list[RunRecord]: + return self._existing(mode, scenario, arm) or self._guarded(fn, mode, scenario, arm, root, exclusion) + + def _blocked(self, mode: str, scenario: str, arm: str, root: str | None, exclusion: str, + notes: str = "S1 failed") -> list[RunRecord]: + """One untested row per arm carrying the current reason: measured rows are kept, a stale + blocked row is replaced, and a repeated sweep never stacks blocked rows.""" + d, existing = self._load(mode, scenario, arm) + if existing and any(r.verdict != "untested" for r in existing): + return existing + if existing and all(r.notes == notes for r in existing): + return existing + shutil.rmtree(d, ignore_errors=True) + return [self.record(mode, scenario, arm, root, exclusion, None, "untested", {}, notes=notes)] + + def record_blocked(self, mode: str, notes: str) -> list[RunRecord]: + native = self.adapter.native_root + out: list[RunRecord] = [] + for scenario, arm, uses_root, exclusion in arms_for(mode): + out += self._blocked(mode, scenario, arm, native if uses_root else None, exclusion, notes=notes) + return out + + def _ask(self, sb: Sandbox, mode: str, prompt: str) -> AskResult: + if mode == "tui": + session = self._open(sb, "tui") + try: + res = self._ask_session(session, mode, prompt) + finally: + session.close() + else: + res = self.adapter.ask(sb, mode, prompt) + return self._checked(res, mode) + + def _prompt(self, mode: str, skill: ProbeSkill, again: bool = False) -> str: + # The screen echoes whatever is typed, so the interactive form asks for a marked line. + return tui_prompt(skill, again) if mode == "tui" else single_prompt(skill, again) + + def _ask_session(self, session: Session, mode: str, prompt: str) -> AskResult: + return self._checked(session.ask(prompt), mode) + + def _checked(self, res: AskResult, mode: str) -> AskResult: + """A run that produced no answer is a failure to measure, not a skill that was not there: + a refusal, a crash and an unparseable stream all leave nothing to judge.""" + if res.reply_text.strip() and not any(m in res.notes for m in ("extract failed:", "timeout")): + return res + where = "screen" if mode == "tui" else "output" + reason = "" + if res.stderr_path and Path(res.stderr_path).is_file(): + text = Path(res.stderr_path).read_text(errors="replace") + errors = [line.strip() for line in text.splitlines() if "Error" in line or "error" in line] + reason = (errors[0] if errors else text[-300:].replace("\n", " | "))[:300] + raise RuntimeError(f"no reply read from the {where}; exit {res.exit_code}; {res.notes}; stderr: {reason}") + + def _open(self, sb: Sandbox, mode: str) -> Session: + session = self.adapter.open_session(sb, mode) + if session is None: + raise RuntimeError(f"no {mode} session for this entry") + return session + + def _judge_turn(self, mode: str, res: AskResult, skill: ProbeSkill) -> str: + return judge_single(skill.token, parse_reply(res.reply_text, self._raw_for(mode, res), skill.name)) + + def _remove_skill(self, sb: Sandbox, root: str, skill: ProbeSkill) -> None: + a = self.adapter + if a.flat_skill_layout: + a.skill_file(sb, root, skill.name).unlink() + else: + shutil.rmtree(a.skill_dir(sb, root, skill.name)) + + @staticmethod + def _raw_for(mode: str, res: AskResult) -> str: + # A screen carries the echoed prompt and any tool panel, so it never stands in for a reply. + return "" if mode == "tui" else res.raw + + def _hook_dict(self, sb: Sandbox, mechanism: str, config_path: str) -> dict: + stamp = read_stamp(stamp_path(sb.config_root)) or {} + return {"mechanism": mechanism, "config_path": config_path, + "fired_at": stamp.get("fired_at"), "fired_at_mtime": stamp.get("fired_at_mtime")} + + # -- arms (each runs inside _guarded, which owns the sandbox's teardown) ------------------ + + def arm_s0(self, mode: str) -> RunRecord: + started = time.time() + sb = self.sandbox() + skill = ProbeSkill.fresh() + res = self._ask(sb, mode, self._prompt(mode, skill)) + try: + verdict = judge_control(parse_reply(res.reply_text, res.raw, skill.name)) + except PromptDesignFailure as ex: + # The reply that broke the control is the evidence for it: record before unwinding. + self.record(mode, "S0", "S0/none", None, "none", res, "untested", {}, sb=sb, + started=started, notes=f"prompt design failure: {ex}", name=skill.name) + raise + return self.record(mode, "S0", "S0/none", None, "none", res, verdict, {}, sb=sb, started=started, + name=skill.name) + + def arm_s1(self, mode: str, exclusion: str = "none", scenario: str = "S1", + root: str | None = None) -> RunRecord: + started = time.time() + a = self.adapter + root = root or a.native_root + sb = self.sandbox() + skill = ProbeSkill.fresh() + write_skill(sb.repo / root, skill, flat=a.flat_skill_layout) + rel = str(a.skill_dir(sb, root, skill.name).relative_to(sb.repo)) + apply_exclusion(sb.repo, exclusion, rel) + arm = "S1/native" if scenario == "S1" else f"S3/{exclusion}" + try: + assert_untracked_state(sb.repo, rel, exclusion) + except AssertionError as ex: + # The arm cannot say anything about this exclusion, so it does not spend a turn. + return self.record(mode, scenario, arm, root, exclusion, None, "untested", + {"native": skill.token}, sb=sb, started=started, + notes=f"git state: {ex}") + res = self._ask(sb, mode, self._prompt(mode, skill)) + verdict = judge_single(skill.token, parse_reply(res.reply_text, self._raw_for(mode, res), skill.name)) + return self.record(mode, scenario, arm, root, exclusion, res, verdict, + {"native": skill.token}, sb=sb, started=started, name=skill.name) + + def arm_s2(self, mode: str, arm: str) -> RunRecord: + started = time.time() + a = self.adapter + root = a.native_root + sb = self.sandbox() + skill = ProbeSkill.fresh() + target = a.skill_file(sb, root, skill.name) + if arm == "hook-adds-skill": + write_skill(sb.repo / root, ProbeSkill.fresh(), flat=a.flat_skill_layout) + if arm == "registration": + info = a.install_registration(sb, target, skill.render()) + if info is None: + return self.record(mode, "S2", "S2/registration", root, "none", None, "untested", {}, + sb=sb, notes="no registration mechanism for this entry", started=started) + reload_used = True + else: + script = write_hook_script(sb.config_root, target, skill.render(), stamp_path(sb.config_root)) + info = a.install_startup_hook(sb, script) + if info is None: + return self.record(mode, "S2", f"S2/{arm}", root, "none", None, "untested", {}, + sb=sb, notes="no startup hook mechanism for this entry", started=started) + reload_used = False + res = self._ask(sb, mode, self._prompt(mode, skill)) + verdict = judge_single(skill.token, parse_reply(res.reply_text, self._raw_for(mode, res), skill.name), + reload_used=reload_used) + hook = self._hook_dict(sb, info.mechanism, info.config_path) + notes = "" if hook["fired_at"] is not None or arm == "registration" else "hook never fired" + target = info.target or target + if not target.exists(): + notes = (notes + " skill file absent after the turn").strip() + verdict = "untested" + return self.record(mode, "S2", f"S2/{arm}", root, "none", res, verdict, {"native": skill.token}, + hook=hook, sb=sb, started=started, notes=notes, name=skill.name) + + def arm_s3(self, mode: str, exclusion: str) -> RunRecord: + return self.arm_s1(mode, exclusion=exclusion, scenario="S3") + + def arm_s4_all(self, mode: str) -> list[RunRecord]: + started = time.time() + a = self.adapter + sb = self.sandbox() + roots = self._s4_roots() + skills = {key: ProbeSkill.fresh() for key in roots} + for key, skill in skills.items(): + write_skill(sb.repo / roots[key], skill, flat=a.flat_skill_layout and roots[key] == a.native_root) + res = self._ask(sb, mode, multi_prompt()) + reply = parse_reply(res.reply_text, res.raw) + found = sorted(key for key, s in skills.items() if s.token in reply.tokens) + leaked = sorted(key for key in found if roots[key] not in a.documented_roots) + notes = f"found={found} leaked={leaked}" + # One turn, one row per root: a single verdict for eleven roots cannot say which leaked. + return [self.record(mode, "S4", "S4/all-roots", root, "none", res, + judge_root(skills[key].token, root in a.documented_roots, reply), + {key: skills[key].token}, sb=sb, started=started, notes=notes, + name=skills[key].name) + for key, root in roots.items()] + + def arm_s4_confirm(self, mode: str, root_key: str, root: str) -> RunRecord: + started = time.time() + a = self.adapter + sb = self.sandbox() + skill = ProbeSkill.fresh() + write_skill(sb.repo / root, skill, flat=a.flat_skill_layout and root == a.native_root) + res = self._ask(sb, mode, self._prompt(mode, skill)) + verdict = judge_root(skill.token, root in a.documented_roots, + parse_reply(res.reply_text, res.raw, skill.name)) + return self.record(mode, "S4", f"S4/confirm-{root_key}", root, "none", res, verdict, + {root_key: skill.token}, sb=sb, started=started, name=skill.name) + + def arm_s5(self, mode: str, arm: str) -> RunRecord: + started = time.time() + a = self.adapter + root = a.native_root + sb = self.sandbox() + skill = ProbeSkill.fresh() + old: str | None = None + used: str | None = None + if arm in ("update", "delete"): + write_skill(sb.repo / root, skill, flat=a.flat_skill_layout) + session = self._open(sb, mode) + try: + first = self._ask_session(session, mode, self._prompt(mode, skill)) + turn1 = self._judge_turn(mode, first, skill) + if arm in ("update", "delete") and turn1 != "visible_first_turn": + return self.record(mode, "S5", f"S5/{arm}", root, "none", first, "untested", {"native": skill.token}, + sb=sb, started=started, notes=f"turn1={turn1}: no baseline to test against", + name=skill.name) + if arm == "add": + write_skill(sb.repo / root, skill, flat=a.flat_skill_layout) + elif arm == "reload": + write_skill(sb.repo / root, skill, flat=a.flat_skill_layout) + used = session.reload() + if used is None: + return self.record(mode, "S5", "S5/reload", root, "none", first, "untested", {"native": skill.token}, + sb=sb, started=started, notes=f"turn1={turn1}; no reload command for this entry", + name=skill.name) + elif arm == "update": + old = skill.token + skill = skill.variant() + write_skill(sb.repo / root, skill, flat=a.flat_skill_layout) + else: + old = skill.token + self._remove_skill(sb, root, skill) + second = self._ask_session(session, mode, self._prompt(mode, skill, again=True)) + finally: + session.close() + reply = parse_reply(second.reply_text, self._raw_for(mode, second), skill.name) + if arm == "delete": + verdict = judge_delete(old, reply) + elif arm == "reload": + verdict = judge_single(skill.token, reply, reload_used=True) + else: + verdict = judge_update(skill.token, old, reply, live=True) + hook = {"mechanism": used, "config_path": "", "fired_at": None, "fired_at_mtime": None} if used else None + expected = {"old": old} if arm == "delete" else {"native": skill.token} + return self.record(mode, "S5", f"S5/{arm}", root, "none", second, verdict, expected, hook=hook, sb=sb, + started=started, notes=f"turn1={turn1}", name=skill.name, prior=first) + + def arm_s6(self, mode: str, arm: str) -> RunRecord: + started = time.time() + a = self.adapter + root = a.native_root + sb = self.sandbox() + old_skill = ProbeSkill.fresh() + write_skill(sb.repo / root, old_skill, flat=a.flat_skill_layout) + target = a.skill_file(sb, root, old_skill.name) + skill = old_skill.variant() if arm == "update" else old_skill + if arm == "update": + script = write_hook_script(sb.config_root, target, skill.render(), stamp_path(sb.config_root)) + else: + gone = a.skill_file(sb, root, old_skill.name) if a.flat_skill_layout \ + else a.skill_dir(sb, root, old_skill.name) + script = write_hook_script(sb.config_root, target, "", stamp_path(sb.config_root), delete=True, + delete_path=gone) + info = a.install_startup_hook(sb, script) + if info is None: + return self.record(mode, "S6", f"S6/{arm}", root, "none", None, "untested", {}, sb=sb, started=started, + notes="no startup hook mechanism for this entry") + res = self._ask(sb, mode, self._prompt(mode, skill)) + reply = parse_reply(res.reply_text, self._raw_for(mode, res), skill.name) + if arm == "update": + verdict = judge_update(skill.token, old_skill.token, reply, live=False) + else: + verdict = judge_delete(old_skill.token, reply) + hook = self._hook_dict(sb, info.mechanism, info.config_path) + notes = "" + if hook["fired_at"] is None: + notes, verdict = "hook never fired", "untested" + elif arm == "update" and (not target.exists() or skill.body_token not in target.read_text()): + notes, verdict = "skill file not rewritten after the turn", "untested" + elif arm == "delete" and target.exists(): + notes, verdict = "skill file still present after the turn", "untested" + expected = {"native": skill.token, "old": old_skill.token} if arm == "update" else {"old": old_skill.token} + return self.record(mode, "S6", f"S6/{arm}", root, "none", res, verdict, expected, hook=hook, sb=sb, + started=started, notes=notes, name=skill.name) + + def arm_s7(self, mode: str, arm: str) -> RunRecord: + started = time.time() + a = self.adapter + root = a.native_root + sb = self.sandbox() + skill = ProbeSkill.fresh() + if not a.can_resume: + return self.record(mode, "S7", f"S7/{arm}", root, "none", None, "untested", {}, sb=sb, started=started, + notes="no resume launch for this entry") + old: str | None = None + if arm == "update": + write_skill(sb.repo / root, skill, flat=a.flat_skill_layout) + first = self._ask(sb, mode, self._prompt(mode, skill)) + turn1 = self._judge_turn(mode, first, skill) + sid = a.session_id(first) + if sid is None: + return self.record(mode, "S7", f"S7/{arm}", root, "none", first, "untested", {"native": skill.token}, + sb=sb, started=started, notes=f"turn1={turn1}; no session id in the first run's output", + name=skill.name) + if arm == "update": + if turn1 != "visible_first_turn": + return self.record(mode, "S7", f"S7/{arm}", root, "none", first, "untested", {"native": skill.token}, + sb=sb, started=started, notes=f"turn1={turn1}: no baseline to test against", + name=skill.name) + old = skill.token + skill = skill.variant() + write_skill(sb.repo / root, skill, flat=a.flat_skill_layout) + res = a.resume(sb, sid, self._prompt(mode, skill, again=True)) + if res is None: + return self.record(mode, "S7", f"S7/{arm}", root, "none", first, "untested", {"native": skill.token}, + sb=sb, started=started, notes=f"turn1={turn1}; no resume launch for this entry", + name=skill.name) + self._checked(res, mode) + verdict = judge_update(skill.token, old, parse_reply(res.reply_text, res.raw, skill.name), live=False) + return self.record(mode, "S7", f"S7/{arm}", root, "none", res, verdict, {"native": skill.token}, sb=sb, + started=started, notes=f"turn1={turn1} session={sid}", name=skill.name, prior=first) + + def arm_s8(self, mode: str, arm: str) -> RunRecord: + started = time.time() + a = self.adapter + root = a.native_root + sb = self.sandbox() + sb.cwd = sb.repo / "sub" / "dir" + sb.cwd.mkdir(parents=True) + (sb.cwd / "README.md").write_text("nested\n") + skill = ProbeSkill.fresh() + tree = sb.repo if arm == "ancestor" else sb.cwd + write_skill(tree / root, skill, flat=a.flat_skill_layout) + res = self._ask(sb, mode, self._prompt(mode, skill)) + verdict = self._judge_turn(mode, res, skill) + return self.record(mode, "S8", f"S8/{arm}", root, "none", res, verdict, {"native": skill.token}, sb=sb, + started=started, notes="cwd=sub/dir", name=skill.name) + + def arm_s9(self, mode: str, arm: str) -> RunRecord: + started = time.time() + a = self.adapter + root = a.native_root + sb = self.sandbox() + wt = add_worktree(sb) + sb.cwd = wt + skill = ProbeSkill.fresh() + tree = wt if arm == "linked-own" else sb.repo + write_skill(tree / root, skill, flat=a.flat_skill_layout) + rel = str(a.skill_dir(sb, root, skill.name).relative_to(sb.repo)) + apply_exclusion(tree, "info-exclude", rel) + try: + assert_untracked_state(tree, rel, "info-exclude") + except AssertionError as ex: + return self.record(mode, "S9", f"S9/{arm}", root, "info-exclude", None, "untested", + {"native": skill.token}, sb=sb, started=started, notes=f"git state: {ex}") + res = self._ask(sb, mode, self._prompt(mode, skill)) + verdict = self._judge_turn(mode, res, skill) + where = "linked" if tree == wt else "main" + return self.record(mode, "S9", f"S9/{arm}", root, "info-exclude", res, verdict, {"native": skill.token}, + sb=sb, started=started, notes=f"cwd={wt.name} skill_in={where}", name=skill.name) + + def arm_s10(self, mode: str) -> RunRecord: + started = time.time() + a = self.adapter + root = a.native_root + sb = self.sandbox() + skill = ProbeSkill.fresh() + target = a.skill_file(sb, root, skill.name) + session = self._open(sb, "daemon") + try: + first = self._ask_session(session, mode, self._prompt(mode, skill)) + turn1 = self._judge_turn("daemon", first, skill) + script = write_hook_script(sb.config_root, target, skill.render(), stamp_path(sb.config_root)) + info = a.install_startup_hook(sb, script) + if info is None: + return self.record(mode, "S10", "S10/hook-from-peer", root, "none", first, "untested", {}, sb=sb, + started=started, notes="no startup hook mechanism for this entry", name=skill.name) + peer = self._ask(sb, "print", self._prompt("print", skill)) + peer_verdict = self._judge_turn("print", peer, skill) + second = self._ask_session(session, mode, self._prompt(mode, skill, again=True)) + finally: + session.close() + hook = self._hook_dict(sb, info.mechanism, info.config_path) + reply = parse_reply(second.reply_text, second.raw, skill.name) + verdict = judge_update(skill.token, None, reply, live=True) + notes = f"turn1={turn1} peer={peer_verdict}" + if hook["fired_at"] is None: + notes, verdict = notes + " hook never fired", "untested" + return self.record(mode, "S10", "S10/hook-from-peer", root, "none", second, verdict, {"native": skill.token}, + hook=hook, sb=sb, started=started, notes=notes, name=skill.name, prior=peer) + + # -- scenarios ---------------------------------------------------------------------------- + + def run_scenario(self, mode: str, scenario: str, arms: list[str] | None = None) -> list[RunRecord]: + if scenario not in SCENARIOS: + raise ValueError(scenario) + if scenario not in MODE_SCENARIOS[mode]: + return [] + gated = scenario not in ("S0", "S1") and not self._s1_ok(mode) + native = self.adapter.native_root + out: list[RunRecord] = [] + if scenario == "S0": + out += self.run_arm(lambda: self.arm_s0(mode), mode, "S0", "S0/none", None, "none") + elif scenario == "S1": + recs = self.run_arm(lambda: self.arm_s1(mode), mode, "S1", "S1/native", native, "none") + self.s1_ok[mode] = combine([r.verdict for r in recs])[0] == "visible_first_turn" + out += recs + elif scenario == "S2": + for arm in arms or arms_of(mode, "S2"): + if gated: + out += self._blocked(mode, "S2", f"S2/{arm}", native, "none") + elif arm == "registration": + out += self._registration(mode) + else: + out += self.run_arm(lambda a=arm: self.arm_s2(mode, a), mode, "S2", f"S2/{arm}", + native, "none") + elif scenario == "S3": + for exclusion in arms or arms_of(mode, "S3"): + if gated: + out += self._blocked(mode, "S3", f"S3/{exclusion}", native, exclusion) + else: + out += self.run_arm(lambda e=exclusion: self.arm_s3(mode, e), mode, "S3", + f"S3/{exclusion}", native, exclusion) + elif scenario == "S4": + if gated: + out += self._blocked(mode, "S4", "S4/all-roots", None, "none") + return out + recs = self._s4_passes(mode) + out += recs + for key, root in self._s4_roots().items(): + rows = [r for r in recs if r.root == root and key in r.expected_tokens] + if rows and all(r.expected_tokens[key] in r.tokens_found for r in rows): + continue + # One confirmation turn per missed root: the multi prompt is what may have + # stopped enumerating, and a second miss adds no evidence. + out += self.run_once(lambda k=key, r=root: self.arm_s4_confirm(mode, k, r), mode, "S4", + f"S4/confirm-{key}", root, "none") + elif scenario in ("S5", "S6", "S7", "S8", "S9"): + fn = {"S5": self.arm_s5, "S6": self.arm_s6, "S7": self.arm_s7, "S8": self.arm_s8, "S9": self.arm_s9}[scenario] + exclusion = "info-exclude" if scenario == "S9" else "none" + for arm in arms or arms_of(mode, scenario): + if gated: + out += self._blocked(mode, scenario, f"{scenario}/{arm}", native, exclusion) + elif arm == "reload" and self.adapter.tui_reload is None: + # Declared, not discovered: asking the first turn would spend a turn to learn + # what the adapter already knows. + out += self._blocked(mode, scenario, f"{scenario}/{arm}", native, exclusion, + notes="no reload command for this entry") + else: + out += self.run_arm(lambda a_=arm: fn(mode, a_), mode, scenario, f"{scenario}/{arm}", native, exclusion) + elif scenario == "S10": + if gated: + out += self._blocked(mode, "S10", "S10/hook-from-peer", native, "none") + elif "print" not in self.adapter.modes: + out += self._blocked(mode, "S10", "S10/hook-from-peer", native, "none", notes="no print mode for the peer") + else: + out += self.run_arm(lambda: self.arm_s10(mode), mode, "S10", "S10/hook-from-peer", native, "none") + else: + raise ValueError(scenario) + return out + + def _s1_ok(self, mode: str) -> bool: + """The gate holds even when a sweep asks for one later scenario: the recorded control + decides, so a harness whose S1 failed never spends turns on the rest.""" + if mode not in self.s1_ok: + d = run_dir(self.outdir, self.adapter.entry, mode, "S1", "S1/native") + recs = load_runs(d) if (d / "run1.json").exists() else [] + if recs: + self.s1_ok[mode] = combine([r.verdict for r in recs])[0] == "visible_first_turn" + return self.s1_ok.get(mode, True) + + def _s4_passes(self, mode: str) -> list[RunRecord]: + per_pass = len(self._s4_roots()) + existing = self._existing(mode, "S4", "S4/all-roots", per_pass=per_pass) + passes = [existing[i:i + per_pass] for i in range(0, len(existing), per_pass)] + identity = (mode, "S4", "S4/all-roots", None, "none") + while len(passes) < max(1, self.runs): + passes.append(self._guarded(lambda: self.arm_s4_all(mode), *identity)) + if len(passes) == 2 and _verdicts_by_root(passes[0]) != _verdicts_by_root(passes[1]): + passes.append(self._guarded(lambda: self.arm_s4_all(mode), *identity)) + return [r for p in passes for r in p] + + def _registration(self, mode: str) -> list[RunRecord]: + identity = (mode, "S2", "S2/registration", self.adapter.native_root, "none") + + def run() -> RunRecord: + return self.arm_s2(mode, "registration") + + recs = list(self._existing(mode, "S2", "S2/registration")) + # An entry without a registration mechanism says so on the first run; nothing to repeat. + while len(recs) < self.runs and not (recs and recs[-1].verdict == "untested"): + recs += self._guarded(run, *identity) + if needs_third_run([r.verdict for r in recs]): + recs += self._guarded(run, *identity) + return recs + + +def free_phase(adapter: Adapter, outdir: Path, mode: str, base: Path | None) -> dict: + sb = new_sandbox(adapter.lever, adapter.real_root(), list(adapter.credential_files), + list(adapter.passthrough_env), dict(adapter.extra_env), base=base) + try: + adapter.prepare(sb) + info = { + "entry": adapter.entry, "binary": adapter.binary_path(), "version": adapter.version(sb.env), + "os": os_label(), "mode": mode, "isolation_lever": adapter.lever, + "credential_files": [c for c in adapter.credential_files if (sb.config_root / c).exists()], + "auth_ok": adapter.check_auth(sb) if adapter.binary_path() else None, + "catalogue": adapter.list_catalogue(sb) if adapter.binary_path() else None, + } + finally: + sb.cleanup() + d = run_dir(outdir, adapter.entry, mode, "free", "free") + d.mkdir(parents=True, exist_ok=True) + (d / "free.json").write_text(json.dumps(info, indent=2) + "\n") + return info + + +def blocked_reason(info: dict) -> str | None: + if info["binary"] is None: + return "binary not installed" + # auth_ok None is "not measurable for this entry", which is not a reason to skip the turns. + return "auth_ok false in isolated root" if info["auth_ok"] is False else None + + +def main(argv: list[str] | None = None) -> int: + p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + p.add_argument("--harness", action="append", choices=sorted(ENTRIES)) + p.add_argument("--mode", default="print", choices=("print", "daemon", "tui")) + p.add_argument("--scenario", action="append", choices=SCENARIOS) + p.add_argument("--turn", action="store_true") + p.add_argument("--runs", type=int, default=2) + p.add_argument("--outdir", type=Path, default=KIT / "out") + p.add_argument("--matrix", type=Path, default=KIT / "matrix.json") + p.add_argument("--keep", action="store_true") + p.add_argument("--rerun", action="store_true") + p.add_argument("--emit", action="store_true") + p.add_argument("--base", type=Path, default=None) + args = p.parse_args(argv) + + if args.emit: + rows = emit_matrix(args.outdir, args.matrix) + print(f"wrote {len(rows)} rows to {args.matrix}") + return 0 + + entries = args.harness or [e for e in ENTRIES if e != "fake"] + completed = aborted = 0 + for name in entries: + adapter = ENTRIES[name]() + if args.mode not in adapter.modes: + print(f"{name}: mode {args.mode} unsupported, skipping") + continue + try: + info = free_phase(adapter, args.outdir, args.mode, args.base) + print(f"{name}: {info['version']} auth_ok={info['auth_ok']}") + if args.turn: + runner = Runner(adapter, args.outdir, runs=args.runs, keep=args.keep, base=args.base, + version=info["version"], rerun=args.rerun) + blocker = blocked_reason(info) + if blocker: + print(f"{name}: {blocker}, turn arms recorded as untested") + for r in runner.record_blocked(args.mode, blocker): + print(f"{name} {args.mode} {r.arm} -> {r.verdict} {r.notes}") + else: + for scenario in args.scenario or MODE_SCENARIOS[args.mode]: + if scenario not in MODE_SCENARIOS[args.mode]: + print(f"{name}: {scenario} n/a in {args.mode} mode") + continue + for r in runner.run_scenario(args.mode, scenario): + print(f"{name} {args.mode} {r.arm} root={r.root} excl={r.exclusion} " + f"-> {r.verdict} {r.notes}") + completed += 1 + except PromptDesignFailure as ex: + aborted += 1 + print(f"{name}: prompt design failure: {ex}") + except Exception as ex: # noqa: BLE001 + aborted += 1 + print(f"{name}: aborted: {ex!r}") + return 1 if aborted else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/docs/probes/2026-09-16-skills-discovery/report.py b/docs/probes/2026-09-16-skills-discovery/report.py new file mode 100644 index 0000000000..f4960b8ed4 --- /dev/null +++ b/docs/probes/2026-09-16-skills-discovery/report.py @@ -0,0 +1,164 @@ +#!/usr/bin/env python3 +"""Render capability-matrix.md from matrix.json: one row per entry, the columns #778 and #962 read. + +Usage: report.py [--matrix FILE] [--out FILE] +""" +from __future__ import annotations + +import argparse +import json +import re +import sys +from collections import defaultdict +from pathlib import Path + +KIT = Path(__file__).resolve().parent + +VISIBLE = ("visible_first_turn", "visible_after_reload") +COLUMNS = ("Entry", "Version tested", "Modes", "Native root", "Roots consumed", "Startup mechanism proven", + "Exclusion preserving load", "Vendor-isolated destination", "Reload path", "Live catalogue", + "Startup rewrite", "Resume", "Nested cwd", "Worktree", "Peer hook", "Interactive", "GUI status", + "Minimum version") + +# GUI launch modes the kit cannot drive; the findings carry a manual procedure for each. +GUI_STATUS = { + "cursor": "Cursor desktop: untested (manual procedure in findings.md)", + "agy": "Antigravity IDE: untested (manual procedure in findings.md)", + "agy-dirlayout": "Antigravity IDE: untested (manual procedure in findings.md)", + "agy-clidir": "Antigravity IDE: untested (manual procedure in findings.md)", + "kiro": "Kiro IDE: untested (manual procedure in findings.md)", +} + + +def _version_key(version: str) -> tuple: + """Order versions by their numbers, so 1.9 comes before 1.10.""" + return tuple(int(p) if p.isdigit() else p for p in re.split(r"(\d+)", version)) + + +def load_rows(path: Path) -> list[dict]: + return json.loads(path.read_text()) + + +def summarise(rows: list[dict]) -> list[dict]: + by_entry: dict[str, list[dict]] = defaultdict(list) + for r in rows: + by_entry[r["entry"]].append(r) + consumed_by: dict[str, set[str]] = defaultdict(set) + for r in rows: + # A root another vendor loads is shared even when that vendor never documented it, so a + # leaked sighting counts against isolation exactly like a documented one. Two entries of + # one vendor are one consumer: they are the same CLI under different configurations. + if r["scenario"] == "S4" and r["verdict"] in VISIBLE + ("leaked",) and r["root"]: + consumed_by[r["root"]].add(r["harness"]) + out = [] + for entry in sorted(by_entry): + rs = by_entry[entry] + versions = sorted({r["version"] for r in rs}) + modes = sorted({r["mode"] for r in rs}) + native = next((r["root"] for r in rs if r["scenario"] == "S1" and r["root"]), "") + roots = sorted({r["root"] for r in rs if r["scenario"] == "S4" and r["verdict"] in VISIBLE and r["root"]}) + leaked = sorted({r["root"] for r in rs if r["scenario"] == "S4" and r["verdict"] == "leaked"}) + mechanisms = sorted({f"{r['mechanism']} ({r['mode']})" for r in rs + if r["scenario"] == "S2" and r["verdict"] in VISIBLE and r["mechanism"]}) + exclusions = sorted({r["exclusion"] for r in rs if r["scenario"] == "S3" and r["verdict"] in VISIBLE}) + reload = sorted({r["mechanism"] for r in rs if r["verdict"] == "visible_after_reload" and r["mechanism"]}) + untested = sorted({r["notes"][:80] for r in rs if r["verdict"] == "untested" and r["scenario"] == "S1"}) + harness = next((r["harness"] for r in rs if r.get("harness")), entry) + isolated = [root for root in roots if consumed_by[root] == {harness}] + s1 = {r["verdict"] for r in rs if r["scenario"] == "S1"} + if s1 & set(VISIBLE): + status = "measured" + elif untested: + status = "untested: " + "; ".join(untested) + elif s1: + status = "S1 failed: the native root's skill was not loaded" + else: + status = "no S1 row" + measured = status == "measured" + proven = {r["version"] for r in rs if r["scenario"] == "S1" and r["verdict"] in VISIBLE} + ran = {r["scenario"] for r in rs if r["verdict"] != "untested"} + modes_ran = {r["mode"] for r in rs if r["verdict"] != "untested"} + + def arm_cells(scenario: str, mode: str | None = None) -> list[str]: + vals: dict[str, set[str]] = {} + for r in rs: + if r["scenario"] != scenario or r["verdict"] == "untested": + continue + if mode is not None and r["mode"] != mode: + continue + vals.setdefault(r["arm"].split("/", 1)[1], set()).add(r["verdict"]) + return [f"{arm}={'/'.join(sorted(v))}" for arm, v in sorted(vals.items())] + + def tui_cells() -> list[str]: + vals: dict[str, set[str]] = {} + for r in rs: + if r["mode"] != "tui" or r["verdict"] == "untested" or r["scenario"] == "S0": + continue + key = r["scenario"] if r["scenario"] == "S1" else r["arm"].split("/", 1)[1] + vals.setdefault(key, set()).add(r["verdict"]) + order = {"S1": 0, "hook-adds-skill": 1, "add": 2, "reload": 3} + return [f"{k}={'/'.join(sorted(v))}" for k, v in sorted(vals.items(), key=lambda kv: order.get(kv[0], 9))] + + def cell(values: list[str], scenario: str, mode: str | None = None) -> str: + # A scenario the entry never ran says so, instead of reading as a measured "none". + if values: + return "; ".join(values) + if not measured: + return "—" + if scenario == "tui": + ran_it = "tui" in modes_ran + else: + ran_it = any(r["scenario"] == scenario and r["verdict"] != "untested" + and (mode is None or r["mode"] == mode) for r in rs) + return "none" if ran_it else "n/a (not run)" + + out.append({ + "Entry": entry, + "Version tested": ", ".join(versions), + "Modes": ", ".join(modes), + "Native root": native, + "Roots consumed": cell([", ".join(roots) + (f" (undocumented: {', '.join(leaked)})" if leaked else "")] + if roots else [], "S4"), + "Startup mechanism proven": cell(mechanisms, "S2"), + "Exclusion preserving load": cell([", ".join(exclusions)] if exclusions else [], "S3"), + "Vendor-isolated destination": cell([", ".join(isolated)] if isolated else [], "S4"), + "Reload path": cell(reload, "S2"), + "Live catalogue": cell(arm_cells("S5", "daemon"), "S5", "daemon"), + "Startup rewrite": cell(arm_cells("S6"), "S6"), + "Resume": cell(arm_cells("S7"), "S7"), + "Nested cwd": cell(arm_cells("S8"), "S8"), + "Worktree": cell(arm_cells("S9"), "S9"), + "Peer hook": cell(arm_cells("S10"), "S10"), + "Interactive": cell(tui_cells(), "tui"), + "GUI status": GUI_STATUS.get(entry, "n/a"), + "Minimum version": min(proven, key=_version_key) if proven else "—", + "_status": status, + }) + return out + + +def render(summary: list[dict]) -> str: + lines = ["# Capability matrix — repo-local skill discovery", "", + "Generated by `report.py` from `matrix.json`; do not edit by hand. A `—` cell means the entry", + "could not run (see its status line below and `findings.md`).", "", + "| " + " | ".join(COLUMNS) + " |", "|" + " -- |" * len(COLUMNS)] + for row in summary: + lines.append("| " + " | ".join(str(row[c]).replace("|", "\\|") for c in COLUMNS) + " |") + lines += ["", "## Status per entry", ""] + for row in summary: + lines.append(f"- `{row['Entry']}`: {row['_status']}") + return "\n".join(lines) + "\n" + + +def main(argv: list[str] | None = None) -> int: + p = argparse.ArgumentParser(description=__doc__) + p.add_argument("--matrix", type=Path, default=KIT / "matrix.json") + p.add_argument("--out", type=Path, default=KIT / "capability-matrix.md") + args = p.parse_args(argv) + args.out.write_text(render(summarise(load_rows(args.matrix)))) + print(f"wrote {args.out}") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/docs/probes/2026-09-16-skills-discovery/selftest.py b/docs/probes/2026-09-16-skills-discovery/selftest.py new file mode 100644 index 0000000000..20ced14dbb --- /dev/null +++ b/docs/probes/2026-09-16-skills-discovery/selftest.py @@ -0,0 +1,1928 @@ +#!/usr/bin/env python3 +"""Self-tests for the skills discovery probe kit. Needs git and Python; no vendor binary.""" +from __future__ import annotations + +import sys +import unittest +from pathlib import Path + +KIT = Path(__file__).resolve().parent +sys.path.insert(0, str(KIT)) + + +class KitLayoutTests(unittest.TestCase): + def test_out_is_ignored(self): + ignore = (KIT / ".gitignore").read_text().splitlines() + self.assertIn("out/", ignore) + self.assertIn("*.stderr.log", ignore) + + +from lib.probe_skill import ( # noqa: E402 + NO_SKILL, TOKEN_RE, ProbeSkill, multi_prompt, parse_reply, single_prompt, write_skill, +) + + +class ProbeSkillTests(unittest.TestCase): + def test_token_only_in_body(self): + s = ProbeSkill.fresh() + text = s.render() + head, _, body = text.partition("\n---\n") + self.assertIn(f"name: {s.name}", head) + self.assertNotIn(s.token, head) + self.assertIn(s.body_token, body) + self.assertNotIn(s.token, single_prompt(s)) + self.assertNotIn(s.token, multi_prompt()) + self.assertIn(s.name, single_prompt(s)) + + def test_write_skill_layouts(self): + import tempfile + s = ProbeSkill.fresh() + with tempfile.TemporaryDirectory() as d: + root = Path(d) / ".claude" / "skills" + f = write_skill(root, s) + self.assertEqual(f, root / s.name / "SKILL.md") + self.assertEqual(f.read_text(), s.render()) + flat = write_skill(Path(d) / "flat", s, flat=True) + self.assertEqual(flat.name, f"{s.name}.md") + + def test_parse_reply(self): + s = ProbeSkill.fresh() + r = parse_reply(f"The token is {s.body_token}.", f'{{"result":"The token is {s.body_token}."}}') + self.assertEqual(r.tokens, frozenset({s.token})) + self.assertFalse(r.skill_named) + r = parse_reply(f"I can see {s.name} but cannot read it", "") + self.assertEqual(r.tokens, frozenset()) + self.assertTrue(r.skill_named) + r = parse_reply(NO_SKILL, NO_SKILL) + self.assertTrue(r.no_skill) + # raw is the fallback only when no reply text was extracted + r = parse_reply("", f"chunk: {s.body_token}") + self.assertEqual(r.tokens, frozenset({s.token})) + r = parse_reply(NO_SKILL, f'{{"tool_result":"{s.body_token}"}}') + self.assertEqual(r.tokens, frozenset()) + # naming some other probe skill is not naming this one + other = ProbeSkill.fresh() + r = parse_reply(f"The skill listed is {other.name}, not {s.name}.", "", name=s.name) + self.assertTrue(r.skill_named) + r = parse_reply(f"Only {other.name} is available.", "", name=s.name) + self.assertFalse(r.skill_named) + + def test_a_marked_prompt_ignores_an_earlier_reply(self): + from lib.probe_skill import extract_tui_reply, tui_prompt + s = ProbeSkill.fresh() + later = tui_prompt(s, again=True) + self.assertIn("PROBE-MARK-", later) + # The transcript a vendor redraws still holds the previous turn's answer. + screen = "PROBE-REPLY: NO-SKILL\n" + later + "\nPROBE-REPLY: " + later.split("PROBE-REPLY: ")[1][:15] + self.assertEqual(extract_tui_reply(screen, later), "") + mark = re.search(r"PROBE-MARK-[0-9a-f]{4}", later).group(0) + answered = screen + f"\nPROBE-REPLY: {mark} {s.body_token}\n" + self.assertEqual(extract_tui_reply(answered, later), f"PROBE-REPLY: {s.body_token}") + self.assertEqual(extract_tui_reply(answered, tui_prompt(s)), "PROBE-REPLY: NO-SKILL") + + def test_variant_and_tui_prompt(self): + from lib.probe_skill import TUI_REPLY_RE, extract_tui_reply, tui_prompt + s = ProbeSkill.fresh() + v = s.variant() + self.assertEqual(v.name, s.name) + self.assertNotEqual(v.token, s.token) + prompt = tui_prompt(s) + self.assertNotIn(s.token, prompt) + self.assertIn(s.name, prompt) + self.assertEqual(extract_tui_reply(prompt), "") + screen = f"> {prompt}\n\n **PROBE-REPLY: {s.body_token}**\n> " + self.assertEqual(extract_tui_reply(screen), f"PROBE-REPLY: {s.body_token}") + self.assertEqual(extract_tui_reply("PROBE-REPLY: NO-SKILL\n"), "PROBE-REPLY: NO-SKILL") + self.assertIsNotNone(TUI_REPLY_RE.search("PROBE-REPLY: NO-SKILL")) + r = parse_reply(extract_tui_reply(screen), "", s.name) + self.assertIn(s.token, r.tokens) + self.assertFalse(r.skill_named) + + +import os # noqa: E402 +import tempfile # noqa: E402 +from unittest import mock # noqa: E402 + +from lib.isolation import ENV_ALLOWLIST, Sandbox, git, new_sandbox # noqa: E402 + + +class IsolationTests(unittest.TestCase): + def test_sandbox_repo_and_lever(self): + with tempfile.TemporaryDirectory() as d: + real = Path(d) / "real" + (real / "nested").mkdir(parents=True) + (real / "auth.json").write_text("{}") + (real / "nested" / "creds").write_text("x") + (real / "secret.db").write_text("no") + with mock.patch.dict(os.environ, {"KCAP_URL": "https://leak.invalid"}): + sb = new_sandbox("CODEX_HOME", real, ["auth.json", "nested/creds", "missing.json"], + base=Path(d)) + try: + self.assertEqual(sb.repo, sb.repo.resolve()) + self.assertEqual(git(sb.repo, "rev-list", "--count", "HEAD").strip(), "1") + self.assertEqual(sb.env["CODEX_HOME"], str(sb.config_root)) + self.assertEqual(sb.env["HOME"], os.environ["HOME"]) + self.assertNotIn("KCAP_URL", sb.env) + self.assertTrue((sb.config_root / "auth.json").exists()) + self.assertTrue((sb.config_root / "nested" / "creds").exists()) + self.assertFalse((sb.config_root / "secret.db").exists()) + for k in sb.env: + self.assertTrue(k in ENV_ALLOWLIST or k in ("HOME", "CODEX_HOME"), k) + finally: + sb.cleanup() + self.assertFalse(sb.root.exists()) + + def test_git_ignores_the_developer_global_config(self): + with tempfile.TemporaryDirectory() as d: + excludes = Path(d) / "excludes" + excludes.write_text(".claude/\n") + cfg = Path(d) / "gitconfig" + cfg.write_text(f"[core]\n\texcludesFile = {excludes}\n") + with mock.patch.dict(os.environ, {"GIT_CONFIG_GLOBAL": str(cfg)}): + sb = new_sandbox("HOME", None, [], base=Path(d)) + try: + rel = ".claude/skills/kcap-probe-abc123" + (sb.repo / rel).mkdir(parents=True) + (sb.repo / rel / "SKILL.md").write_text("x") + self.assertIn("??", assert_untracked_state(sb.repo, rel, "none")) + finally: + sb.cleanup() + + def test_home_lever_and_passthrough(self): + with tempfile.TemporaryDirectory() as d: + with mock.patch.dict(os.environ, {"GH_TOKEN": "t", "OTHER": "o"}): + sb = new_sandbox("HOME", None, [], passthrough_env=["GH_TOKEN"], + extra_env={"XDG_CONFIG_HOME": "/x"}, base=Path(d)) + try: + self.assertEqual(sb.env["HOME"], str(sb.config_root)) + self.assertEqual(sb.env["GH_TOKEN"], "t") + self.assertNotIn("OTHER", sb.env) + self.assertEqual(sb.env["XDG_CONFIG_HOME"], "/x") + finally: + sb.cleanup() + + def test_cwd_defaults_to_repo_and_worktree_is_linked(self): + from lib.isolation import add_worktree + sb = new_sandbox("FAKE_HOME", None, []) + try: + self.assertEqual(sb.cwd, sb.repo) + wt = add_worktree(sb) + self.assertTrue((wt / ".git").is_file()) + self.assertEqual(git(wt, "branch", "--show-current").strip(), "wt-b") + self.assertEqual(git(sb.repo, "branch", "--show-current").strip(), "main") + self.assertTrue((wt / "README.md").is_file()) + finally: + sb.cleanup() + + +from lib.git_exclusion import EXCLUSIONS, apply, assert_untracked_state # noqa: E402 + + +class GitExclusionTests(unittest.TestCase): + def _sandbox(self, d): + return new_sandbox("HOME", None, [], base=Path(d)) + + def test_none_shows_untracked(self): + with tempfile.TemporaryDirectory() as d: + sb = self._sandbox(d) + (sb.repo / ".claude/skills/kcap-probe-abc123").mkdir(parents=True) + (sb.repo / ".claude/skills/kcap-probe-abc123/SKILL.md").write_text("x") + self.assertIsNone(apply(sb.repo, "none", ".claude/skills/kcap-probe-abc123")) + out = assert_untracked_state(sb.repo, ".claude/skills/kcap-probe-abc123", "none") + self.assertIn("??", out) + with self.assertRaises(AssertionError): + assert_untracked_state(sb.repo, ".claude/skills/kcap-probe-abc123", "gitignore") + with self.assertRaises(AssertionError): + assert_untracked_state(sb.repo, ".claude/skills/kcap-probe-absent", "none") + (sb.repo / ".claude/skills/kcap-probe-empty").mkdir() + with self.assertRaises(AssertionError): + assert_untracked_state(sb.repo, ".claude/skills/kcap-probe-empty", "none") + + def test_gitignore_and_info_exclude(self): + for exclusion in ("gitignore", "info-exclude"): + with tempfile.TemporaryDirectory() as d: + sb = self._sandbox(d) + rel = ".agents/skills/kcap-probe-abc123" + (sb.repo / rel).mkdir(parents=True) + (sb.repo / rel / "SKILL.md").write_text("x") + written = apply(sb.repo, exclusion, rel) + self.assertIsNotNone(written) + self.assertIn(f"/{rel}/", written.read_text()) + out = assert_untracked_state(sb.repo, rel, exclusion) + self.assertEqual(out, "") + if exclusion == "gitignore": + self.assertEqual(git(sb.repo, "status", "--porcelain").strip(), "") + else: + self.assertNotIn(".gitignore", git(sb.repo, "ls-files")) + + def test_info_exclude_inside_worktree(self): + with tempfile.TemporaryDirectory() as d: + sb = self._sandbox(d) + wt = sb.root / "wt" + git(sb.repo, "worktree", "add", "-q", "-b", "wt", str(wt)) + rel = ".pi/skills/kcap-probe-abc123" + (wt / rel).mkdir(parents=True) + (wt / rel / "SKILL.md").write_text("x") + written = apply(wt, "info-exclude", rel) + self.assertEqual(written, written.resolve()) + self.assertEqual(assert_untracked_state(wt, rel, "info-exclude"), "") + self.assertIn(EXCLUSIONS[2], "info-exclude") + + def test_info_exclude_in_linked_worktree_resolves_to_common_dir(self): + from lib.isolation import add_worktree + sb = new_sandbox("FAKE_HOME", None, []) + try: + wt = add_worktree(sb) + d = wt / ".x" / "skills" / "kcap-probe-abc" + d.mkdir(parents=True) + (d / "SKILL.md").write_text("x\n") + target = apply(wt, "info-exclude", ".x/skills/kcap-probe-abc") + self.assertEqual(target, (sb.repo / ".git" / "info" / "exclude").resolve()) + assert_untracked_state(wt, ".x/skills/kcap-probe-abc", "info-exclude") + finally: + sb.cleanup() + + +from lib.probe_skill import Reply # noqa: E402 +from lib.verdict import ( # noqa: E402 + VERDICTS, PromptDesignFailure, combine, judge_control, judge_root, judge_single, needs_third_run, +) + + +class VerdictTests(unittest.TestCase): + def test_judge_single(self): + found = Reply(frozenset({"a" * 12}), True, False) + self.assertEqual(judge_single("a" * 12, found), "visible_first_turn") + self.assertEqual(judge_single("a" * 12, found, reload_used=True), "visible_after_reload") + self.assertEqual(judge_single("b" * 12, Reply(frozenset(), True, False)), "catalogue_only") + self.assertEqual(judge_single("b" * 12, Reply(frozenset(), False, True)), "not_visible") + # the name echoed beside an explicit NO-SKILL is a negative, not a sighting + self.assertEqual(judge_single("b" * 12, Reply(frozenset(), True, True)), "not_visible") + + def test_judge_control(self): + self.assertEqual(judge_control(Reply(frozenset(), False, True)), "not_visible") + with self.assertRaises(PromptDesignFailure): + judge_control(Reply(frozenset({"c" * 12}), False, False)) + + def test_judge_root(self): + r = Reply(frozenset({"d" * 12}), True, False) + self.assertEqual(judge_root("d" * 12, True, r), "visible_first_turn") + self.assertEqual(judge_root("d" * 12, False, r), "leaked") + self.assertEqual(judge_root("e" * 12, True, r), "not_visible") + + def test_combine(self): + self.assertEqual(combine([]), ("untested", False)) + self.assertEqual(combine(["visible_first_turn", "visible_first_turn"]), ("visible_first_turn", False)) + self.assertEqual(combine(["visible_first_turn", "not_visible", "visible_first_turn"]), + ("visible_first_turn", True)) + self.assertTrue(needs_third_run(["visible_first_turn", "not_visible"])) + self.assertFalse(needs_third_run(["not_visible", "not_visible"])) + self.assertFalse(needs_third_run(["not_visible"])) + for v in VERDICTS: + self.assertEqual(combine([v]), (v, False)) + + def test_judge_update_and_delete(self): + from lib.verdict import judge_delete, judge_update + new, old = "a" * 12, "b" * 12 + r = lambda text: parse_reply(text, text, "kcap-probe-abcdef") # noqa: E731 + self.assertEqual(judge_update(new, old, r(f"PROBE-BODY-{new}"), live=True), "visible_live") + self.assertEqual(judge_update(new, old, r(f"PROBE-BODY-{new}"), live=False), "visible_first_turn") + self.assertEqual(judge_update(new, old, r(f"PROBE-BODY-{old}"), live=True), "stale") + self.assertEqual(judge_update(new, None, r("NO-SKILL"), live=True), "not_visible") + self.assertEqual(judge_update(new, None, r("kcap-probe-abcdef is listed but empty"), live=True), "catalogue_only") + self.assertEqual(judge_delete(old, r("NO-SKILL")), "revoked") + self.assertEqual(judge_delete(old, r(f"PROBE-BODY-{old}")), "stale") + self.assertEqual(judge_delete(old, r("kcap-probe-abcdef exists but I cannot read it")), "stale") + self.assertEqual(judge_delete(old, r("I have no idea")), "not_visible") + for v in ("visible_live", "stale", "revoked"): + self.assertIn(v, VERDICTS) + + +import json # noqa: E402 + +from lib.recorder import RunRecord, emit_matrix, load_runs, os_label, run_dir, write_run # noqa: E402 + + +def _rec(**over): + base = dict( + entry="fake", harness="fake", binary="/bin/fake", version="1.0", os=os_label(), mode="print", + argv=["fake", "-p"], isolation_lever="FAKE_HOME", credential_files=[], auth_ok=True, + scenario="S1", arm="S1/native", root=".fake/skills", exclusion="none", + hook=None, first_request_at=1.0, reply="tok", tokens_found=["a" * 12], skill_named=True, + stderr_path=None, verdict="visible_first_turn", duration_ms=10, expected_tokens={"native": "a" * 12}, + ) + base.update(over) + return RunRecord(**base) + + +class RecorderTests(unittest.TestCase): + def test_write_load_emit(self): + with tempfile.TemporaryDirectory() as d: + out = Path(d) / "out" + p1 = write_run(out, _rec()) + p2 = write_run(out, _rec(verdict="not_visible")) + self.assertEqual(p1.name, "run1.json") + self.assertEqual(p2.name, "run2.json") + self.assertEqual(p1.parent, run_dir(out, "fake", "print", "S1", "S1/native")) + write_run(out, _rec(verdict="visible_first_turn")) + write_run(out, _rec(version="0.9", verdict="not_visible", arm="S0/none", scenario="S0")) + self.assertEqual(len(load_runs(out)), 4) + rows = emit_matrix(out, Path(d) / "matrix.json") + self.assertEqual(len(rows), 2) + by_version = {r["version"]: r for r in rows} + row = by_version["1.0"] + self.assertEqual(row["verdict"], "visible_first_turn") + self.assertTrue(row["flaky"]) + self.assertEqual(row["runs"], 3) + self.assertEqual(row["arm"], "S1/native") + self.assertTrue(all(e.startswith("out/") for e in row["evidence"])) + older = by_version["0.9"] + self.assertEqual(older["runs"], 1) + self.assertEqual(older["verdict"], "not_visible") + self.assertEqual(older["arm"], "S0/none") + self.assertEqual(json.loads((Path(d) / "matrix.json").read_text())[0]["entry"], "fake") + + def test_os_label(self): + self.assertRegex(os_label(), r"^\S+ \S+ \S+$") + + +import subprocess # noqa: E402 +import time # noqa: E402 + +from lib.hook_script import read_stamp, stamp_path, write_hook_script # noqa: E402 + + +class HookScriptTests(unittest.TestCase): + def test_script_writes_skill_and_stamp(self): + with tempfile.TemporaryDirectory() as d: + cfg = Path(d) / "cfg" + cfg.mkdir() + skill = ProbeSkill.fresh() + target = Path(d) / "repo" / ".claude" / "skills" / skill.name / "SKILL.md" + stamp = stamp_path(cfg) + script = write_hook_script(cfg, target, skill.render(), stamp) + self.assertEqual(script.stat().st_mode & 0o111, 0o111) + self.assertIsNone(read_stamp(stamp)) + proc = subprocess.run([str(script)], input='{"hook_event_name":"SessionStart"}', + capture_output=True, text=True, timeout=10) + self.assertEqual(proc.returncode, 0, proc.stderr) + self.assertEqual(proc.stdout, "") + self.assertEqual(target.read_text(), skill.render()) + stamped = read_stamp(stamp) + self.assertIn("fired_at", stamped) + self.assertIsInstance(stamped["fired_at_mtime"], float) + self.assertAlmostEqual(stamped["fired_at_mtime"], stamp.stat().st_mtime, places=3) + self.assertEqual(Path(str(stamp) + ".stdin").read_text(), '{"hook_event_name":"SessionStart"}') + + def test_script_tolerates_a_closed_stdin(self): + with tempfile.TemporaryDirectory() as d: + cfg = Path(d) / "cfg" + cfg.mkdir() + skill = ProbeSkill.fresh() + target = Path(d) / "repo" / ".x" / skill.name / "SKILL.md" + script = write_hook_script(cfg, target, skill.render(), stamp_path(cfg)) + proc = subprocess.run(["sh", "-c", f"exec 0<&-; '{script}'"], capture_output=True, text=True, timeout=15) + self.assertEqual(proc.returncode, 0, proc.stderr) + self.assertEqual(proc.stderr, "") + self.assertEqual(target.read_text(), skill.render()) + self.assertIsNotNone(read_stamp(stamp_path(cfg))) + + def test_script_survives_open_stdin_and_can_delete(self): + with tempfile.TemporaryDirectory() as d: + cfg = Path(d) / "cfg" + cfg.mkdir() + skill = ProbeSkill.fresh() + target = Path(d) / "repo" / ".x" / skill.name / "SKILL.md" + target.parent.mkdir(parents=True) + target.write_text("old") + stamp = stamp_path(cfg) + script = write_hook_script(cfg, target, "", stamp, delete=True) + with subprocess.Popen([str(script)], stdin=subprocess.PIPE, stdout=subprocess.PIPE) as p: + out, _ = p.communicate(timeout=10) + self.assertEqual(p.returncode, 0) + self.assertFalse(target.parent.exists()) + self.assertIsNotNone(read_stamp(stamp)) + + +from harness.base import Adapter, AskResult, HookInfo, Session # noqa: E402 +from harness.fake import FakeAdapter # noqa: E402 +from lib.print_driver import print_ask # noqa: E402 + + +class HookOnATerminalTests(unittest.TestCase): + def test_hook_survives_a_closed_stdin(self): + with tempfile.TemporaryDirectory() as d: + root = Path(d) + script = write_hook_script(root, root / "skills" / "x" / "SKILL.md", "body\n", stamp_path(root)) + # `exec` is a special builtin: a redirection it cannot perform would end the shell. + out = subprocess.run(["/bin/sh", "-c", "exec 0<&- ; '" + str(script) + "'"], + capture_output=True, text=True, timeout=20) + self.assertEqual(out.returncode, 0, out.stderr) + self.assertEqual(out.stderr.strip(), "") + self.assertTrue((root / "skills" / "x" / "SKILL.md").exists()) + + def test_delete_removes_the_skill_not_the_root(self): + with tempfile.TemporaryDirectory() as d: + root = Path(d) + flat = root / "skills" / "kcap-probe-abc.md" + flat.parent.mkdir(parents=True) + flat.write_text("body\n") + script = write_hook_script(root, flat, "", stamp_path(root), delete=True, delete_path=flat) + subprocess.run([str(script)], input="{}", capture_output=True, text=True, timeout=20) + self.assertFalse(flat.exists()) + # A flat layout shares its root with every other skill the vendor owns. + self.assertTrue(flat.parent.is_dir()) + + + def test_hook_leaves_a_terminal_alone(self): + import pty as _pty + with tempfile.TemporaryDirectory() as d: + root = Path(d) + script = write_hook_script(root, root / "skills" / "x" / "SKILL.md", "body\n", stamp_path(root)) + master, slave = _pty.openpty() + try: + started = time.time() + out = subprocess.run([str(script)], stdin=slave, stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, timeout=20) + elapsed = time.time() - started + finally: + os.close(master) + os.close(slave) + self.assertEqual(out.returncode, 0) + self.assertTrue((root / "skills" / "x" / "SKILL.md").exists()) + # A terminal's input belongs to the vendor's UI, so the hook neither reads nor waits. + self.assertFalse((Path(str(stamp_path(root)) + ".stdin")).exists()) + self.assertLess(elapsed, 2.0) + + +class AdapterTests(unittest.TestCase): + def test_fake_adapter_round_trip(self): + with tempfile.TemporaryDirectory() as d: + fa = FakeAdapter() + sb = new_sandbox(fa.lever, None, [], base=Path(d)) + skill = ProbeSkill.fresh() + write_skill(sb.repo / fa.native_root, skill) + res = fa.ask(sb, "print", single_prompt(skill)) + self.assertIsInstance(res, AskResult) + self.assertIn(skill.body_token, res.reply_text) + self.assertIn(skill.body_token, res.raw) + self.assertEqual(fa.skill_file(sb, fa.native_root, skill.name), + sb.repo / ".fake/skills" / skill.name / "SKILL.md") + + def test_fake_adapter_runs_hook_before_reading(self): + with tempfile.TemporaryDirectory() as d: + fa = FakeAdapter() + sb = new_sandbox(fa.lever, None, [], base=Path(d)) + skill = ProbeSkill.fresh() + script = write_hook_script(sb.config_root, fa.skill_file(sb, fa.native_root, skill.name), + skill.render(), stamp_path(sb.config_root)) + info = fa.install_startup_hook(sb, script) + self.assertIsInstance(info, HookInfo) + res = fa.ask(sb, "print", single_prompt(skill)) + self.assertIn(skill.body_token, res.reply_text) + self.assertIsNotNone(read_stamp(stamp_path(sb.config_root))) + + def test_print_driver_timeout_and_extract(self): + with tempfile.TemporaryDirectory() as d: + res = print_ask(["sh", "-c", "echo '{\"result\":\"hi\"}'"], Path(d), dict(os.environ), + Path(d) / "e.stderr.log", timeout=10, + extract=lambda raw: json.loads(raw)["result"]) + self.assertEqual(res.reply_text, "hi") + self.assertEqual(res.exit_code, 0) + slow = print_ask(["sh", "-c", "sleep 5"], Path(d), dict(os.environ), Path(d) / "s.stderr.log", timeout=0.5) + self.assertIsNone(slow.exit_code) + + def test_print_driver_reports_extraction_failure(self): + with tempfile.TemporaryDirectory() as d: + def boom(raw): + raise ValueError("not json") + + res = print_ask(["sh", "-c", "echo not-json"], Path(d), dict(os.environ), + Path(d) / "x.stderr.log", timeout=10, extract=boom) + self.assertTrue(res.notes.startswith("extract failed"), res.notes) + self.assertEqual(res.reply_text, "not-json\n") + + def test_adapter_defaults(self): + self.assertIsNone(Adapter.check_auth(FakeAdapter(), None)) + self.assertIsNone(Adapter.install_registration(FakeAdapter(), None, None, None)) + self.assertIsNone(Adapter.install_startup_hook(FakeAdapter(), None, None)) + self.assertEqual(Adapter.credential_files, ()) + self.assertEqual(Adapter.passthrough_env, ()) + with self.assertRaises(TypeError): + Adapter.extra_env["leak"] = "1" + + def test_fake_session_resume_and_defaults(self): + from harness.base import Adapter, Session + a = FakeAdapter() + self.assertFalse(Adapter.can_resume) + self.assertIsNone(Adapter().open_session(None, "daemon")) + sb = new_sandbox("FAKE_HOME", None, []) + try: + skill = ProbeSkill.fresh() + s = a.open_session(sb, "daemon") + self.assertIsInstance(s, Session) + self.assertIn(NO_SKILL, s.ask(single_prompt(skill)).reply_text) + write_skill(sb.repo / ".fake" / "skills", skill) + self.assertIn(skill.body_token, s.ask(single_prompt(skill)).reply_text) + s.close() + frozen = a.__class__() + frozen.live_catalogue = False + fs = frozen.open_session(sb, "daemon") + other = ProbeSkill.fresh() + write_skill(sb.repo / ".fake" / "skills", other) + self.assertNotIn(other.body_token, fs.ask(single_prompt(other)).reply_text) + self.assertEqual(fs.reload(), "/reload") + self.assertIn(other.body_token, fs.ask(single_prompt(other)).reply_text) + fs.close() + res = a.ask(sb, "print", single_prompt(skill)) + self.assertEqual(a.session_id(res), "fake-session") + self.assertTrue(a.can_resume) + resumed = a.resume(sb, "fake-session", single_prompt(skill)) + self.assertIn(skill.body_token, resumed.reply_text) + self.assertIn("--resume", resumed.argv) + finally: + sb.cleanup() + + +import probe # noqa: E402 +from lib.recorder import load_runs as _load # noqa: E402 + + +class _NoFireAdapter(FakeAdapter): + """A hook that is installed but never runs, so the skill file never lands.""" + + def install_startup_hook(self, sb, script): + return HookInfo(mechanism="fake-startup", config_path=str(script)) + + +class _CountingAdapter(FakeAdapter): + def __init__(self): + super().__init__() + self.calls = 0 + + def ask(self, sb, mode, prompt): + self.calls += 1 + return super().ask(sb, mode, prompt) + + +class _FrozenAdapter(FakeAdapter): + live_catalogue = False + + +class _AncestorAdapter(FakeAdapter): + """A vendor anchored at the git root: reads the repo's roots wherever it is launched.""" + + def catalogue(self, sb): + lines = [] + for root in self.read_roots: + for skill_md in sorted((sb.repo / root).glob("kcap-probe-*/SKILL.md")): + m = TOKEN_RE.search(skill_md.read_text()) + if m: + lines.append(f"{skill_md.parent.name}=PROBE-BODY-{m.group(1)}") + return lines + + +class _NoResumeAdapter(_CountingAdapter): + can_resume = False + + +class _RaisingAdapter(FakeAdapter): + def ask(self, sb, mode, prompt): + raise RuntimeError("vendor exploded") + + +class _NoHookAdapter(FakeAdapter): + def install_startup_hook(self, sb, script): + return None + + +class _LeakyReaderAdapter(FakeAdapter): + """Reads .agents/skills without documenting it.""" + + documented_roots = frozenset({".fake/skills"}) + + +class _PreIgnoredAdapter(FakeAdapter): + """The native root is already ignored, so the none arm cannot prove the path is inside the repo.""" + + def __init__(self): + super().__init__() + self.calls = 0 + + def prepare(self, sb): + (sb.repo / ".gitignore").write_text(".fake/skills/\n") + git(sb.repo, "add", ".gitignore") + git(sb.repo, "commit", "-q", "-m", "ignore the skills root") + + def ask(self, sb, mode, prompt): + self.calls += 1 + return super().ask(sb, mode, prompt) + + +class _StderrAdapter(FakeAdapter): + def ask(self, sb, mode, prompt): + log = sb.root / "fake.stderr.log" + log.write_text("vendor noise\n") + res = super().ask(sb, mode, prompt) + res.stderr_path = str(log) + return res + + +class _NoBinaryAdapter(FakeAdapter): + entry = "nobin" + + def binary_path(self): + return None + + +class _RaisingStderrAdapter(FakeAdapter): + """The vendor wrote its complaint to stderr before the driver gave up.""" + + def ask(self, sb, mode, prompt): + (sb.root / "fake.stderr.log").write_text("boom\n") + raise RuntimeError("vendor exploded") + + +class _FailingVendorAdapter(FakeAdapter): + """The vendor exited non-zero without answering; its stderr carries the reason.""" + + def ask(self, sb, mode, prompt): + log = sb.root / "fake.stderr.log" + log.write_text("IneligibleTierError: this client is no longer supported\n") + return AskResult(reply_text="", raw="", argv=["fake"], started_at=0.0, first_request_at=0.0, + stderr_path=str(log), exit_code=41) + + +class _RaisingFreePhaseAdapter(FakeAdapter): + """The vendor could not even be inspected.""" + + def prepare(self, sb): + raise RuntimeError("no such vendor") + + +class _UnparseableAdapter(FakeAdapter): + """The vendor printed something its adapter could not read as an answer.""" + + def ask(self, sb, mode, prompt): + body = '{"stop_reason": "refusal", "is_error": true}' + return AskResult(reply_text=body, raw=body, argv=["fake"], started_at=0.0, first_request_at=0.0, + stderr_path=None, exit_code=0, notes="extract failed: ValueError('is_error')") + + +class _OddLogNameAdapter(FakeAdapter): + """A driver that names its stderr file outside the *.stderr.log pattern.""" + + def ask(self, sb, mode, prompt): + log = sb.root / "vendor.log" + log.write_text("odd noise\n") + res = super().ask(sb, mode, prompt) + res.stderr_path = str(log) + return res + + +class _LeakyControlAdapter(FakeAdapter): + """Every reply carries a token, so S0 cannot act as a negative control.""" + + def __init__(self): + super().__init__() + self.skill = ProbeSkill.fresh() + + def ask(self, sb, mode, prompt): + text = self.skill.body_token + return AskResult(reply_text=text, raw=text, argv=["fake"], started_at=0.0, + first_request_at=0.0, stderr_path=None, exit_code=0) + + +class RunnerTests(unittest.TestCase): + def _runner(self, d, adapter=None, runs=1): + return probe.Runner(adapter or FakeAdapter(), Path(d) / "out", runs=runs, base=Path(d)) + + def test_s0_and_s1(self): + with tempfile.TemporaryDirectory() as d: + r = self._runner(d, runs=2) + recs = r.run_scenario("print", "S0") + self.assertEqual([x.verdict for x in recs], ["not_visible", "not_visible"]) + recs = r.run_scenario("print", "S1") + self.assertEqual([x.verdict for x in recs], ["visible_first_turn"] * 2) + self.assertTrue(r.s1_ok["print"]) + self.assertEqual(recs[0].root, ".fake/skills") + self.assertEqual(recs[0].expected_tokens.keys(), {"native"}) + self.assertEqual(len(_load(Path(d) / "out")), 4) + + def test_s2_hook_arms_and_registration_skip(self): + with tempfile.TemporaryDirectory() as d: + r = self._runner(d) + r.run_scenario("print", "S1") + recs = r.run_scenario("print", "S2") + by_arm = {} + for x in recs: + by_arm.setdefault(x.arm, []).append(x) + self.assertEqual({x.verdict for x in by_arm["S2/hook-creates-root"]}, {"visible_first_turn"}) + self.assertEqual({x.verdict for x in by_arm["S2/hook-adds-skill"]}, {"visible_first_turn"}) + self.assertEqual({x.verdict for x in by_arm["S2/registration"]}, {"untested"}) + self.assertIsNotNone(by_arm["S2/hook-creates-root"][0].hook["fired_at"]) + self.assertIsNotNone(by_arm["S2/hook-creates-root"][0].hook["fired_at_mtime"]) + self.assertEqual(by_arm["S2/hook-creates-root"][0].hook["mechanism"], "fake-startup") + + def test_s3_exclusions(self): + with tempfile.TemporaryDirectory() as d: + r = self._runner(d) + r.run_scenario("print", "S1") + recs = r.run_scenario("print", "S3") + self.assertEqual({x.exclusion for x in recs}, {"gitignore", "info-exclude"}) + self.assertEqual({x.verdict for x in recs}, {"visible_first_turn"}) + + def test_s4_roots_and_confirmation(self): + with tempfile.TemporaryDirectory() as d: + r = self._runner(d, runs=2) + r.run_scenario("print", "S1") + recs = r.run_scenario("print", "S4") + all_rows = [x for x in recs if x.arm == "S4/all-roots"] + expected_keys = set(probe.ALL_ROOTS) | {"native_fake"} + self.assertEqual(len(all_rows), 2 * len(expected_keys)) + self.assertEqual({next(iter(x.expected_tokens)) for x in all_rows}, expected_keys) + self.assertEqual({len(x.expected_tokens) for x in all_rows}, {1}) + per_root = {} + for x in all_rows: + per_root.setdefault(x.root, set()).add(x.verdict) + self.assertEqual(len(per_root), len(expected_keys)) + self.assertEqual(per_root[".fake/skills"], {"visible_first_turn"}) + self.assertEqual(per_root[".agents/skills"], {"visible_first_turn"}) + self.assertEqual(per_root[".claude/skills"], {"not_visible"}) + self.assertIn("found=['agents', 'native_fake'] leaked=[]", all_rows[0].notes) + confirms = [x for x in recs if x.arm.startswith("S4/confirm-")] + self.assertEqual(len(confirms), len(expected_keys) - 2) + self.assertEqual(len({x.arm for x in confirms}), len(confirms)) + by_root = {x.root: x.verdict for x in confirms} + self.assertEqual(by_root[".claude/skills"], "not_visible") + self.assertNotIn(".fake/skills", by_root) + self.assertNotIn(".agents/skills", by_root) + + def test_s4_leaked_root_gets_its_own_row(self): + with tempfile.TemporaryDirectory() as d: + r = probe.Runner(_LeakyReaderAdapter(), Path(d) / "out", runs=1, base=Path(d)) + recs = r.run_scenario("print", "S4") + rows = {x.root: x for x in recs if x.arm == "S4/all-roots"} + self.assertEqual(rows[".agents/skills"].verdict, "leaked") + self.assertEqual(rows[".fake/skills"].verdict, "visible_first_turn") + self.assertIn("leaked=['agents']", rows[".agents/skills"].notes) + self.assertNotIn(".agents/skills", {x.root for x in recs if x.arm.startswith("S4/confirm-")}) + + def test_s1_skips_the_turn_when_the_skill_is_already_ignored(self): + with tempfile.TemporaryDirectory() as d: + a = _PreIgnoredAdapter() + r = probe.Runner(a, Path(d) / "out", runs=2, base=Path(d)) + recs = r.run_scenario("print", "S1") + self.assertEqual([x.verdict for x in recs], ["untested", "untested"]) + self.assertIn("git state:", recs[0].notes) + self.assertEqual(a.calls, 0) + self.assertFalse(r.s1_ok["print"]) + + def test_s2_without_a_startup_hook_is_untested(self): + with tempfile.TemporaryDirectory() as d: + r = probe.Runner(_NoHookAdapter(), Path(d) / "out", runs=1, base=Path(d)) + recs = r.run_scenario("print", "S2", arms=["hook-creates-root"]) + self.assertEqual([x.verdict for x in recs], ["untested"]) + self.assertEqual(recs[0].notes, "no startup hook mechanism for this entry") + + def test_s1_gate_blocks_later_scenarios(self): + with tempfile.TemporaryDirectory() as d: + r = self._runner(d) + r.s1_ok["print"] = False + recs = r.run_scenario("print", "S3") + self.assertEqual([x.verdict for x in recs], ["untested", "untested"]) + self.assertEqual(recs[0].notes, "S1 failed") + + def test_s2_hook_never_fired_is_untested(self): + with tempfile.TemporaryDirectory() as d: + r = probe.Runner(_NoFireAdapter(), Path(d) / "out", runs=2, base=Path(d)) + r.run_scenario("print", "S1") + recs = r.run_scenario("print", "S2", arms=["hook-creates-root"]) + self.assertEqual({x.verdict for x in recs}, {"untested"}) + for x in recs: + self.assertIn("hook never fired", x.notes) + self.assertIn("skill file absent after the turn", x.notes) + + def test_stderr_log_survives_sandbox_cleanup(self): + with tempfile.TemporaryDirectory() as d: + out = Path(d) / "out" + r = probe.Runner(_StderrAdapter(), out, runs=1, base=Path(d)) + recs = r.run_scenario("print", "S1") + copied = out / "fake" / "print" / "S1" / "S1_native" / "run1.fake.stderr.log" + self.assertEqual(copied.read_text(), "vendor noise\n") + self.assertEqual(recs[0].stderr_path, str(copied)) + written = json.loads((copied.parent / "run1.json").read_text()) + self.assertEqual(written["stderr_path"], str(copied)) + + def test_arm_exception_is_untested(self): + with tempfile.TemporaryDirectory() as d: + r = probe.Runner(_RaisingAdapter(), Path(d) / "out", runs=2, base=Path(d)) + recs = r.run_scenario("print", "S1") + self.assertEqual([x.verdict for x in recs], ["untested", "untested"]) + for x in recs: + self.assertIn("exception=RuntimeError", x.notes) + self.assertEqual(len(_load(Path(d) / "out")), 2) + + def test_completed_arm_resumes_and_rerun_clears(self): + with tempfile.TemporaryDirectory() as d: + out = Path(d) / "out" + a = _CountingAdapter() + first = probe.Runner(a, out, runs=2, base=Path(d)).run_scenario("print", "S1") + self.assertEqual(a.calls, 2) + again = probe.Runner(a, out, runs=2, base=Path(d)).run_scenario("print", "S1") + self.assertEqual(a.calls, 2) + self.assertEqual([x.reply for x in again], [x.reply for x in first]) + self.assertEqual(len(_load(out)), 2) + probe.Runner(a, out, runs=2, base=Path(d), rerun=True).run_scenario("print", "S1") + self.assertEqual(a.calls, 4) + self.assertEqual(len(_load(out)), 2) + + def test_untested_arm_is_rerun_not_latched(self): + with tempfile.TemporaryDirectory() as d: + out = Path(d) / "out" + probe.Runner(_RaisingAdapter(), out, runs=2, base=Path(d)).run_scenario("print", "S1") + self.assertEqual({x.verdict for x in _load(out)}, {"untested"}) + a = _CountingAdapter() + recs = probe.Runner(a, out, runs=2, base=Path(d)).run_scenario("print", "S1") + self.assertEqual([x.verdict for x in recs], ["visible_first_turn"] * 2) + self.assertEqual(a.calls, 2) + self.assertEqual(len(_load(out)), 2) + + def test_partial_arm_is_completed_on_resume(self): + with tempfile.TemporaryDirectory() as d: + out = Path(d) / "out" + a = _CountingAdapter() + probe.Runner(a, out, runs=1, base=Path(d)).run_scenario("print", "S1") + self.assertEqual(a.calls, 1) + recs = probe.Runner(a, out, runs=2, base=Path(d)).run_scenario("print", "S1") + self.assertEqual(a.calls, 2) + self.assertEqual(len(recs), 2) + self.assertEqual(len(_load(out)), 2) + + def test_gated_rows_are_idempotent(self): + with tempfile.TemporaryDirectory() as d: + out = Path(d) / "out" + for _ in range(2): + r = probe.Runner(FakeAdapter(), out, runs=2, base=Path(d)) + r.s1_ok["print"] = False + recs = r.run_scenario("print", "S3") + self.assertEqual([x.verdict for x in recs], ["untested", "untested"]) + self.assertEqual(len(_load(out)), 2) + + def test_blocked_rows_carry_the_current_reason(self): + with tempfile.TemporaryDirectory() as d: + out = Path(d) / "out" + first = probe.Runner(FakeAdapter(), out, runs=2, base=Path(d)) + first.record_blocked("print", "binary not installed") + second = probe.Runner(FakeAdapter(), out, runs=2, base=Path(d)) + second.s1_ok["print"] = False + recs = second.run_scenario("print", "S3") + self.assertEqual({x.notes for x in recs}, {"S1 failed"}) + self.assertEqual(len([r for r in _load(out) if r.scenario == "S3"]), 2) + + def test_exception_path_keeps_stderr_and_tears_down(self): + with tempfile.TemporaryDirectory() as d: + out = Path(d) / "out" + recs = probe.Runner(_RaisingStderrAdapter(), out, runs=1, base=Path(d)).run_scenario("print", "S1") + copied = out / "fake" / "print" / "S1" / "S1_native" / "run1.fake.stderr.log" + self.assertEqual(copied.read_text(), "boom\n") + self.assertEqual(recs[0].stderr_path, str(copied)) + self.assertEqual([p.name for p in Path(d).iterdir() if p.name.startswith("skprobe-")], []) + + def test_failed_vendor_run_is_untested_with_its_stderr(self): + with tempfile.TemporaryDirectory() as d: + out = Path(d) / "out" + recs = probe.Runner(_FailingVendorAdapter(), out, runs=1, base=Path(d)).run_scenario("print", "S1") + self.assertEqual(recs[0].verdict, "untested") + self.assertIn("no reply read from the output; exit 41", recs[0].notes) + self.assertIn("IneligibleTierError", recs[0].notes) + self.assertTrue(Path(recs[0].stderr_path).exists()) + + def test_protocol_failure_without_a_reply_is_untested(self): + class _ProtocolFailure(FakeAdapter): + def ask(self, sb, mode, prompt): + return AskResult(reply_text="", raw="[]", argv=["fake", "acp"], started_at=0.0, first_request_at=0.0, + stderr_path=None, exit_code=None, notes='session/new failed: {"error": "no key"}') + with tempfile.TemporaryDirectory() as d: + recs = probe.Runner(_ProtocolFailure(), Path(d) / "out", runs=1, base=Path(d)).run_scenario("print", "S1") + self.assertEqual(recs[0].verdict, "untested") + self.assertIn("session/new failed", recs[0].notes) + + def test_interactive_log_is_kept_when_the_screen_failed(self): + with tempfile.TemporaryDirectory() as d: + out = Path(d) / "out" + a = FakeAdapter() + + class _NoAnswerSession(Session): + def __init__(self, sb): + (sb.root / "fake-tui.log").write_text("the screen as it was\n") + + def ask(self, prompt): + return AskResult(reply_text="", raw="", argv=["fake"], started_at=0.0, first_request_at=0.0, + stderr_path=None, exit_code=None, notes="timeout") + + def close(self): + return None + + a.open_session = lambda sb, mode: _NoAnswerSession(sb) + recs = probe.Runner(a, out, runs=1, base=Path(d)).run_scenario("tui", "S1") + self.assertEqual([x.verdict for x in recs], ["untested"]) + # The screen is the only evidence a failed interactive turn leaves. + kept = out / "fake" / "tui" / "S1" / "S1_native" / "run1.fake-tui.log" + self.assertEqual(kept.read_text(), "the screen as it was\n") + + def test_odd_log_name_is_still_kept(self): + with tempfile.TemporaryDirectory() as d: + out = Path(d) / "out" + recs = probe.Runner(_OddLogNameAdapter(), out, runs=1, base=Path(d)).run_scenario("print", "S1") + copied = out / "fake" / "print" / "S1" / "S1_native" / "run1.vendor.log" + self.assertEqual(copied.read_text(), "odd noise\n") + self.assertEqual(recs[0].stderr_path, str(copied)) + + def test_s4_partial_pass_is_redone(self): + with tempfile.TemporaryDirectory() as d: + out = Path(d) / "out" + a = _CountingAdapter() + probe.Runner(a, out, runs=1, base=Path(d)).run_scenario("print", "S4") + arm_dir = out / "fake" / "print" / "S4" / "S4_all-roots" + files = sorted(arm_dir.glob("run*.json")) + self.assertEqual(len(files), len(probe.ALL_ROOTS) + 1) + for f in files[1:]: + f.unlink() + calls = a.calls + probe.Runner(a, out, runs=1, base=Path(d)).run_scenario("print", "S4") + self.assertEqual(len(list(arm_dir.glob("run*.json"))), len(files)) + self.assertEqual(a.calls, calls + 1) + + def test_s0_prompt_design_failure_keeps_evidence(self): + with tempfile.TemporaryDirectory() as d: + out = Path(d) / "out" + a = _LeakyControlAdapter() + r = probe.Runner(a, out, runs=2, base=Path(d)) + with self.assertRaises(PromptDesignFailure): + r.run_scenario("print", "S0") + recs = _load(out) + self.assertEqual(len(recs), 1) + self.assertEqual(recs[0].verdict, "untested") + self.assertIn(a.skill.body_token, recs[0].reply) + self.assertIn("prompt design failure", recs[0].notes) + + def test_third_run_on_disagreement(self): + with tempfile.TemporaryDirectory() as d: + r = self._runner(d, runs=2) + calls = {"n": 0} + + def flaky(): + calls["n"] += 1 + v = "visible_first_turn" if calls["n"] != 2 else "not_visible" + return probe.RunRecord(entry="fake", harness="fake", binary="x", version="1", os="o", + mode="print", argv=[], isolation_lever="L", credential_files=[], + auth_ok=None, scenario="S1", arm="S1/native", root=None, + exclusion="none", hook=None, first_request_at=None, reply="", + tokens_found=[], skill_named=False, stderr_path=None, verdict=v, + duration_ms=0) + recs = r.run_arm(flaky, "print", "S1", "S1/native", None, "none") + self.assertEqual(len(recs), 3) + + def test_s5_live_catalogue(self): + with tempfile.TemporaryDirectory() as d: + recs = self._runner(d).run_scenario("daemon", "S5") + self.assertEqual({r.arm: r.verdict for r in recs}, + {"S5/add": "visible_live", "S5/update": "visible_live", "S5/delete": "revoked"}) + self.assertTrue(all("turn1=" in r.notes for r in recs)) + self.assertTrue((Path(d) / "out" / "fake" / "daemon" / "S5" / "S5_add" / "run1.prior.raw.txt").is_file()) + + def test_s5_frozen_catalogue_is_stale(self): + with tempfile.TemporaryDirectory() as d: + recs = self._runner(d, _FrozenAdapter()).run_scenario("daemon", "S5") + self.assertEqual({r.arm: r.verdict for r in recs}, + {"S5/add": "not_visible", "S5/update": "stale", "S5/delete": "stale"}) + + def test_s5_without_a_session_is_untested_without_a_turn(self): + with tempfile.TemporaryDirectory() as d: + a = _CountingAdapter() + a.open_session = lambda sb, mode: None + recs = self._runner(d, a).run_scenario("daemon", "S5") + self.assertEqual({r.verdict for r in recs}, {"untested"}) + self.assertEqual(a.calls, 0) + + def test_s6_startup_mutation(self): + with tempfile.TemporaryDirectory() as d: + recs = self._runner(d).run_scenario("print", "S6") + self.assertEqual({r.arm: r.verdict for r in recs}, {"S6/update": "visible_first_turn", "S6/delete": "revoked"}) + self.assertTrue(all(r.hook and r.hook["fired_at"] for r in recs)) + with tempfile.TemporaryDirectory() as d: + recs = self._runner(d, _NoFireAdapter()).run_scenario("print", "S6") + self.assertEqual({r.verdict for r in recs}, {"untested"}) + self.assertTrue(all("hook never fired" in r.notes for r in recs)) + + def test_s7_resume(self): + with tempfile.TemporaryDirectory() as d: + recs = self._runner(d).run_scenario("print", "S7") + self.assertEqual({r.arm: r.verdict for r in recs}, {"S7/add": "visible_first_turn", "S7/update": "visible_first_turn"}) + self.assertTrue(all("session=fake-session" in r.notes and "--resume" in r.argv for r in recs)) + with tempfile.TemporaryDirectory() as d: + a = _NoResumeAdapter() + recs = self._runner(d, a).run_scenario("print", "S7") + self.assertEqual({r.verdict for r in recs}, {"untested"}) + self.assertEqual(a.calls, 0) + + def test_s8_nested_cwd(self): + with tempfile.TemporaryDirectory() as d: + recs = self._runner(d).run_scenario("print", "S8") + self.assertEqual({r.arm: r.verdict for r in recs}, {"S8/ancestor": "not_visible", "S8/local": "visible_first_turn"}) + self.assertTrue(all("cwd=sub/dir" in r.notes for r in recs)) + with tempfile.TemporaryDirectory() as d: + recs = self._runner(d, _AncestorAdapter()).run_scenario("print", "S8") + self.assertEqual({r.arm: r.verdict for r in recs}, {"S8/ancestor": "visible_first_turn", "S8/local": "not_visible"}) + + def test_s9_worktrees(self): + with tempfile.TemporaryDirectory() as d: + recs = self._runner(d).run_scenario("print", "S9") + self.assertEqual({r.arm: r.verdict for r in recs}, + {"S9/linked-own": "visible_first_turn", "S9/linked-other": "not_visible"}) + self.assertTrue(all(r.exclusion == "info-exclude" for r in recs)) + + def test_s10_peer_hook(self): + with tempfile.TemporaryDirectory() as d: + recs = self._runner(d).run_scenario("daemon", "S10") + self.assertEqual([(r.arm, r.verdict) for r in recs], [("S10/hook-from-peer", "visible_live")]) + self.assertIn("peer=visible_first_turn", recs[0].notes) + self.assertIn("turn1=not_visible", recs[0].notes) + + def test_tui_mode_runs_controls_hook_and_live_arms(self): + with tempfile.TemporaryDirectory() as d: + r = self._runner(d) + self.assertEqual([x.verdict for x in r.run_scenario("tui", "S0")], ["not_visible"]) + self.assertEqual([x.verdict for x in r.run_scenario("tui", "S1")], ["visible_first_turn"]) + recs = r.run_scenario("tui", "S2") + self.assertEqual({x.arm: x.verdict for x in recs}, {"S2/hook-adds-skill": "visible_first_turn"}) + recs = r.run_scenario("tui", "S5") + self.assertEqual({x.arm: x.verdict for x in recs}, {"S5/add": "visible_live", "S5/reload": "visible_after_reload"}) + self.assertEqual([x.hook["mechanism"] for x in recs if x.arm == "S5/reload"], ["/reload"]) + with tempfile.TemporaryDirectory() as d: + recs = self._runner(d, _FrozenAdapter()).run_scenario("tui", "S5") + self.assertEqual({x.arm: x.verdict for x in recs}, {"S5/add": "not_visible", "S5/reload": "visible_after_reload"}) + + def test_tui_asks_for_the_marked_line_and_untested_without_one(self): + with tempfile.TemporaryDirectory() as d, mock.patch.dict(os.environ, {"KCAP_FAKE_TUI_BARE": "1"}): + # A vendor that answers with a bare token is unreadable on a screen that echoes the + # prompt: that is a failure to measure, not a skill that was not there. + recs = self._runner(d).run_scenario("tui", "S1") + self.assertEqual([x.verdict for x in recs], ["untested"]) + self.assertEqual(recs[0].tokens_found, []) + self.assertIn("no reply read from the screen", recs[0].notes) + + def test_recorded_row_reads_the_screen_under_the_same_rule_as_the_verdict(self): + with tempfile.TemporaryDirectory() as d: + r = self._runner(d) + skill = ProbeSkill.fresh() + res = AskResult(reply_text="", raw=f"tool panel showed {skill.body_token}", argv=[], + started_at=0.0, first_request_at=0.0, stderr_path=None, exit_code=None) + tui = r.record("tui", "S1", "S1/native", ".fake/skills", "none", res, "not_visible", {}, + name=skill.name) + self.assertEqual(tui.tokens_found, []) + printed = r.record("print", "S1", "S1/native", ".fake/skills", "none", res, "not_visible", {}, + name=skill.name) + self.assertEqual(printed.tokens_found, [skill.token]) + + def test_an_answerless_run_is_untested_not_a_negative(self): + with tempfile.TemporaryDirectory() as d: + recs = self._runner(d, _UnparseableAdapter()).run_scenario("print", "S1") + self.assertEqual([x.verdict for x in recs], ["untested"]) + self.assertIn("no reply read from the output", recs[0].notes) + self.assertEqual(recs[0].tokens_found, []) + + def test_a_later_turn_asks_for_the_token_the_skill_carries_now(self): + with tempfile.TemporaryDirectory() as d: + r = self._runner(d) + skill = ProbeSkill.fresh() + self.assertNotIn("again", r._prompt("print", skill)) + self.assertIn("again", r._prompt("print", skill, again=True)) + self.assertIn("PROBE-REPLY", r._prompt("tui", skill, again=True)) + + def test_reload_arm_costs_nothing_without_a_reload_command(self): + with tempfile.TemporaryDirectory() as d: + a = _CountingAdapter() + a.tui_reload = None + recs = self._runner(d, a).run_scenario("tui", "S5", arms=["reload"]) + self.assertEqual([x.verdict for x in recs], ["untested"]) + self.assertIn("no reload command", recs[0].notes) + self.assertEqual(a.calls, 0) + + def test_a_later_scenario_still_stands_behind_the_control(self): + with tempfile.TemporaryDirectory() as d: + out = Path(d) / "out" + # A failed control recorded by an earlier sweep still gates a scenario asked for alone. + a = _CountingAdapter() + probe.Runner(a, out, runs=1, base=Path(d)).run_scenario("print", "S1") + arm = out / "fake" / "print" / "S1" / "S1_native" + data = json.loads((arm / "run1.json").read_text()) + data["verdict"] = "not_visible" + (arm / "run1.json").write_text(json.dumps(data)) + fresh = _CountingAdapter() + recs = probe.Runner(fresh, out, runs=1, base=Path(d)).run_scenario("print", "S3") + self.assertEqual({x.verdict for x in recs}, {"untested"}) + self.assertTrue(all("S1 failed" in x.notes for x in recs)) + self.assertEqual(fresh.calls, 0) + + def test_one_aborted_harness_fails_the_sweep(self): + with tempfile.TemporaryDirectory() as d, mock.patch.dict(probe.ENTRIES, {"fake": _RaisingFreePhaseAdapter}): + ok = probe.main(["--harness", "fake", "--mode", "print", "--outdir", str(Path(d) / "out"), + "--base", str(d)]) + # A sweep that could not run one of its harnesses did not succeed. + self.assertEqual(ok, 1) + + def test_mode_scenario_table_and_blocked_rows(self): + with tempfile.TemporaryDirectory() as d: + r = self._runner(d) + self.assertEqual(r.run_scenario("print", "S5"), []) + self.assertEqual(r.run_scenario("tui", "S4"), []) + self.assertEqual({x.scenario for x in r.record_blocked("tui", "nope")}, {"S0", "S1", "S2", "S5"}) + self.assertEqual({x.scenario for x in r.record_blocked("daemon", "nope")}, + {"S0", "S1", "S2", "S3", "S4", "S5", "S6", "S10"}) + self.assertEqual({x.scenario for x in r.record_blocked("print", "nope")}, + {"S0", "S1", "S2", "S3", "S4", "S6", "S7", "S8", "S9"}) + + def test_cli_records_untested_rows_without_a_binary(self): + with tempfile.TemporaryDirectory() as d: + out = Path(d) / "out" + with mock.patch.dict(probe.ENTRIES, {"nobin": _NoBinaryAdapter}): + code = probe.main(["--harness", "nobin", "--mode", "print", "--turn", + "--outdir", str(out), "--base", d]) + self.assertEqual(code, 0) + recs = _load(out) + self.assertEqual(len(recs), 16) + with mock.patch.dict(probe.ENTRIES, {"nobin": _NoBinaryAdapter}): + probe.main(["--harness", "nobin", "--mode", "print", "--turn", + "--outdir", str(out), "--base", d]) + self.assertEqual(len(_load(out)), 16) + self.assertEqual({r.verdict for r in recs}, {"untested"}) + self.assertEqual({r.notes for r in recs}, {"binary not installed"}) + self.assertEqual({r.arm for r in recs}, { + "S0/none", "S1/native", "S2/hook-creates-root", "S2/hook-adds-skill", "S2/registration", + "S3/gitignore", "S3/info-exclude", "S4/all-roots", "S6/update", "S6/delete", "S7/add", + "S7/update", "S8/ancestor", "S8/local", "S9/linked-own", "S9/linked-other"}) + + def test_cli_stops_the_entry_on_a_prompt_design_failure(self): + with tempfile.TemporaryDirectory() as d: + out = Path(d) / "out" + with mock.patch.dict(probe.ENTRIES, {"leaky": _LeakyControlAdapter}): + code = probe.main(["--harness", "leaky", "--mode", "print", "--turn", "--scenario", "S0", + "--outdir", str(out), "--base", d]) + self.assertEqual(code, 1) + self.assertEqual([r.verdict for r in _load(out)], ["untested"]) + + def test_cli_free_phase_and_emit(self): + with tempfile.TemporaryDirectory() as d: + out = Path(d) / "out" + code = probe.main(["--harness", "fake", "--mode", "print", "--outdir", str(out), "--base", d]) + self.assertEqual(code, 0) + self.assertTrue((out / "fake" / "print" / "free" / "free" / "free.json").exists()) + code = probe.main(["--harness", "fake", "--mode", "print", "--scenario", "S1", "--turn", + "--outdir", str(out), "--base", d, "--matrix", str(Path(d) / "m.json")]) + self.assertEqual(code, 0) + code = probe.main(["--emit", "--outdir", str(out), "--matrix", str(Path(d) / "m.json")]) + self.assertEqual(code, 0) + rows = json.loads((Path(d) / "m.json").read_text()) + self.assertEqual({r["scenario"] for r in rows}, {"S1"}) + + +from lib.acp_driver import acp_ask # noqa: E402 + +SERVERS = KIT / "selftest_servers.py" + + +def _fake_repo(d: str, skill: ProbeSkill | None) -> Path: + repo = Path(d) / "repo" + if skill is not None: + write_skill(repo / ".fake" / "skills", skill) + else: + repo.mkdir(parents=True, exist_ok=True) + return repo + + +class AcpDriverTests(unittest.TestCase): + def test_one_turn(self): + with tempfile.TemporaryDirectory() as d: + skill = ProbeSkill.fresh() + repo = _fake_repo(d, skill) + res = acp_ask([sys.executable, str(SERVERS), "acp"], repo, dict(os.environ), single_prompt(skill), + Path(d) / "acp.stderr.log", timeout=30) + self.assertIn(skill.body_token, res.reply_text) + self.assertIn("stopReason=end_turn", res.notes) + self.assertIn("tools_used=0", res.notes) + self.assertGreaterEqual(res.first_request_at, res.started_at) + self.assertEqual(res.argv[-1], "acp") + + def test_grandchild_holding_stdout_does_not_wedge_shutdown(self): + with tempfile.TemporaryDirectory() as d: + skill = ProbeSkill.fresh() + repo = _fake_repo(d, skill) + env = dict(os.environ, KCAP_FAKE_GRANDCHILD="1") + started = time.time() + res = acp_ask([sys.executable, str(SERVERS), "acp"], repo, env, single_prompt(skill), + Path(d) / "acp.stderr.log", timeout=30) + self.assertIn(skill.body_token, res.reply_text) + self.assertLess(time.time() - started, 20) + + def test_tool_call_updates_are_counted(self): + with tempfile.TemporaryDirectory() as d: + skill = ProbeSkill.fresh() + repo = _fake_repo(d, skill) + env = dict(os.environ, KCAP_FAKE_TOOL_CALL="1") + res = acp_ask([sys.executable, str(SERVERS), "acp"], repo, env, single_prompt(skill), + Path(d) / "acp.stderr.log", timeout=30) + self.assertIn("tools_used=1", res.notes) + self.assertIn(skill.body_token, res.reply_text) + + def test_session_takes_two_turns(self): + from lib.acp_driver import AcpSession + with tempfile.TemporaryDirectory() as d: + skill = ProbeSkill.fresh() + repo = _fake_repo(d, None) + s = AcpSession([sys.executable, str(SERVERS), "acp"], repo, dict(os.environ), Path(d) / "acp.stderr.log", timeout=30) + s.start() + try: + first = s.ask(single_prompt(skill)) + self.assertIn(NO_SKILL, first.reply_text) + write_skill(repo / ".fake" / "skills", skill) + second = s.ask(single_prompt(skill)) + self.assertIn(skill.body_token, second.reply_text) + self.assertNotIn(skill.body_token, first.reply_text) + self.assertGreater(second.first_request_at, first.first_request_at) + self.assertIn("tools_used=0", second.notes) + self.assertIn("initialize", first.raw) + self.assertNotIn("initialize", second.raw) + self.assertEqual(second.raw.count("session/prompt"), 1) + finally: + s.close() + + +from lib.appserver_driver import appserver_ask, hook_state_override # noqa: E402 +from lib.pirpc_driver import pirpc_ask # noqa: E402 +from lib.jsonl_child import JsonlChild # noqa: E402 + + +class JsonlDriversTests(unittest.TestCase): + def test_jsonl_child_round_trip(self): + with tempfile.TemporaryDirectory() as d: + c = JsonlChild([sys.executable, str(SERVERS), "pirpc"], Path(d), dict(os.environ), Path(d) / "c.stderr.log") + c.start() + c.send({"id": "1", "type": "prompt", "message": "x", "streamingBehavior": "followUp"}) + first = c.recv(10) + self.assertEqual(first["type"], "response") + c.close_stdin() + c.stop() + self.assertGreaterEqual(len(c.frames), 2) + + def test_appserver_turn(self): + with tempfile.TemporaryDirectory() as d: + skill = ProbeSkill.fresh() + repo = _fake_repo(d, skill) + res = appserver_ask(str(SERVERS), repo, dict(os.environ), single_prompt(skill), + Path(d) / "as.stderr.log", timeout=30) + self.assertIn(skill.body_token, res.reply_text) + self.assertIn("hook_trust=none", res.notes) + self.assertIn("tools_used=0", res.notes) + self.assertEqual(res.argv[:2], [str(SERVERS), "app-server"]) + + def test_pirpc_turn(self): + with tempfile.TemporaryDirectory() as d: + skill = ProbeSkill.fresh() + repo = _fake_repo(d, skill) + res = pirpc_ask([sys.executable, str(SERVERS), "pirpc"], repo, dict(os.environ), single_prompt(skill), + Path(d) / "pi.stderr.log", timeout=30) + self.assertIn(skill.body_token, res.reply_text) + self.assertIn("tools_used=0", res.notes) + + def test_tool_events_are_counted(self): + with tempfile.TemporaryDirectory() as d: + skill = ProbeSkill.fresh() + repo = _fake_repo(d, skill) + env = dict(os.environ, KCAP_FAKE_TOOL_CALL="1") + served = appserver_ask(str(SERVERS), repo, env, single_prompt(skill), + Path(d) / "as.stderr.log", timeout=30) + self.assertIn("tools_used=1", served.notes) + self.assertIn(skill.body_token, served.reply_text) + piped = pirpc_ask([sys.executable, str(SERVERS), "pirpc"], repo, env, single_prompt(skill), + Path(d) / "pi.stderr.log", timeout=30) + self.assertIn("tools_used=1", piped.notes) + self.assertIn(skill.body_token, piped.reply_text) + + def test_hook_state_override(self): + untrusted = [{"key": "/x/hooks.json:SessionStart:0:0", "trustStatus": "untrusted", + "currentHash": "sha256:abc"}] + self.assertEqual(hook_state_override(untrusted), + 'hooks.state={"/x/hooks.json:SessionStart:0:0"={trusted_hash="sha256:abc"}}') + trusted = [{"key": "/x/hooks.json:SessionStart:0:0", "trustStatus": "trusted", + "currentHash": "sha256:abc"}] + self.assertIsNone(hook_state_override(trusted)) + no_hash = [{"key": "/x/hooks.json:SessionStart:0:0", "trustStatus": "untrusted"}] + self.assertIsNone(hook_state_override(no_hash)) + + def test_appserver_seeds_hook_trust(self): + with tempfile.TemporaryDirectory() as d: + skill = ProbeSkill.fresh() + repo = _fake_repo(d, skill) + env = dict(os.environ) + env["KCAP_FAKE_UNTRUSTED_HOOK"] = "1" + res = appserver_ask(str(SERVERS), repo, env, single_prompt(skill), Path(d) / "seed.stderr.log", timeout=30) + self.assertIn("hook_trust=seeded", res.notes) + idx = res.argv.index("-c") + self.assertTrue(res.argv[idx + 1].startswith("hooks.state=")) + self.assertIn(skill.body_token, res.reply_text) + self.assertIn("hooks/list", res.raw) + + def test_grandchild_holding_stdout_does_not_wedge_the_driver(self): + with tempfile.TemporaryDirectory() as d: + skill = ProbeSkill.fresh() + repo = _fake_repo(d, skill) + env = dict(os.environ, KCAP_FAKE_GRANDCHILD="1") + started = time.time() + res = pirpc_ask([sys.executable, str(SERVERS), "pirpc"], repo, env, single_prompt(skill), + Path(d) / "pi.stderr.log", timeout=30) + self.assertIn(skill.body_token, res.reply_text) + self.assertLess(time.time() - started, 20) + + def test_spawn_failure_is_reported(self): + with tempfile.TemporaryDirectory() as d: + res = pirpc_ask(["/nonexistent/binary"], Path(d), dict(os.environ), "x", + Path(d) / "sf.stderr.log", timeout=5) + self.assertIsNone(res.exit_code) + self.assertIn("exception=", res.notes) + + def test_appserver_session_takes_two_turns(self): + from lib.appserver_driver import AppServerSession + with tempfile.TemporaryDirectory() as d: + skill = ProbeSkill.fresh() + repo = _fake_repo(d, None) + s = AppServerSession(str(SERVERS), repo, dict(os.environ), Path(d) / "as.stderr.log", timeout=30) + s.start() + try: + first = s.ask(single_prompt(skill)) + self.assertIn(NO_SKILL, first.reply_text) + write_skill(repo / ".fake" / "skills", skill) + second = s.ask(single_prompt(skill)) + self.assertIn(skill.body_token, second.reply_text) + self.assertIn("thread/start", first.raw) + self.assertNotIn("thread/start", second.raw) + self.assertEqual(second.raw.count("turn/start"), 1) + finally: + s.close() + + def test_pirpc_session_takes_two_turns(self): + from lib.pirpc_driver import PiRpcSession + with tempfile.TemporaryDirectory() as d: + skill = ProbeSkill.fresh() + repo = _fake_repo(d, None) + s = PiRpcSession([sys.executable, str(SERVERS), "pirpc"], repo, dict(os.environ), Path(d) / "pi.stderr.log", timeout=30) + s.start() + try: + self.assertIn(NO_SKILL, s.ask(single_prompt(skill)).reply_text) + write_skill(repo / ".fake" / "skills", skill) + second = s.ask(single_prompt(skill)) + self.assertIn(skill.body_token, second.reply_text) + self.assertEqual(second.raw.count('"type": "prompt"'), 1) + finally: + s.close() + + +from lib.pty_driver import PtySession, strip_ansi # noqa: E402 +from lib.probe_skill import tui_prompt # noqa: E402 + + +class PtyDriverTests(unittest.TestCase): + def _session(self, d, skill=None, env=None, **kw): + repo = _fake_repo(d, skill) + argv = [sys.executable, str(SERVERS), "tui"] + s = PtySession(argv, repo, dict(os.environ, **(env or {})), Path(d) / "fake-tui.log", ready_idle=0.3, + timeout=15, **kw) + s.start() + return repo, s + + def test_strip_ansi(self): + self.assertEqual(strip_ansi("\x1b[1mbold\x1b[0m\r\n\x1b]0;title\x07x"), "bold\nx") + + def test_a_split_control_sequence_survives(self): + from lib.pty_driver import Terminal + whole = "line\r\n\x1b[2;1H\x1b[KPROBE-REPLY: NO-SKILL\x1b]0;title\x07 tail" + want = Terminal(6, 60) + want.feed(whole) + for cut in range(1, len(whole)): + with self.subTest(cut=cut): + # The operating system splits a read wherever it likes, including mid-sequence. + t = Terminal(6, 60) + t.feed(whole[:cut]) + t.feed(whole[cut:]) + self.assertEqual(t.text(), want.text()) + self.assertIn("PROBE-REPLY: NO-SKILL tail", want.text()) + + def test_ask_reads_the_reply_not_the_echoed_prompt(self): + with tempfile.TemporaryDirectory() as d: + skill = ProbeSkill.fresh() + repo, s = self._session(d, skill) + try: + res = s.ask(tui_prompt(skill)) + self.assertEqual(res.reply_text, f"PROBE-REPLY: {skill.body_token}") + self.assertIn("PROBE-REPLY: ", res.raw) + self.assertGreaterEqual(res.first_request_at, res.started_at) + other = ProbeSkill.fresh() + self.assertEqual(s.ask(tui_prompt(other)).reply_text, "PROBE-REPLY: NO-SKILL") + finally: + s.close() + self.assertIsNotNone(s.proc.poll()) + self.assertTrue((Path(d) / "fake-tui.log").stat().st_size > 0) + + def test_dialog_is_answered_before_the_prompt(self): + with tempfile.TemporaryDirectory() as d: + skill = ProbeSkill.fresh() + repo, s = self._session(d, skill, env={"KCAP_FAKE_TUI_DIALOG": "1"}, + dialogs=((r"trust this folder", "y\r"),)) + try: + res = s.ask(tui_prompt(skill)) + self.assertEqual(res.reply_text, f"PROBE-REPLY: {skill.body_token}") + self.assertIn("dialog=", res.notes) + self.assertNotIn("dialog=", s.ask(tui_prompt(skill)).notes) + finally: + s.close() + + def test_a_failed_start_leaves_no_child(self): + with tempfile.TemporaryDirectory() as d: + # A vendor that draws nothing never becomes ready, and nobody holds the session to close. + s = PtySession([sys.executable, "-c", "import time; time.sleep(60)"], Path(d), dict(os.environ), + Path(d) / "t.log", ready_idle=0.3, timeout=2) + with self.assertRaises(TimeoutError): + s.start() + self.assertIsNotNone(s.proc.poll()) + self.assertTrue(s._log.closed) + + def test_spawn_failure_releases_the_terminal(self): + with tempfile.TemporaryDirectory() as d: + s = PtySession(["/nonexistent/kcap-probe-binary"], Path(d), dict(os.environ), Path(d) / "t.log", + ready_idle=0.3, timeout=2) + with self.assertRaises(OSError): + s.start() + s.close() + self.assertTrue(s._log.closed) + with self.assertRaises(OSError): + os.fstat(s.master) + + def test_reload_refreshes_a_frozen_catalogue(self): + with tempfile.TemporaryDirectory() as d: + skill = ProbeSkill.fresh() + repo, s = self._session(d, None, env={"KCAP_FAKE_TUI_FROZEN": "1"}, reload_command="/reload") + try: + self.assertEqual(s.ask(tui_prompt(skill)).reply_text, "PROBE-REPLY: NO-SKILL") + write_skill(repo / ".fake" / "skills", skill) + self.assertEqual(s.ask(tui_prompt(skill)).reply_text, "PROBE-REPLY: NO-SKILL") + self.assertEqual(s.reload(), "/reload") + self.assertEqual(s.ask(tui_prompt(skill)).reply_text, f"PROBE-REPLY: {skill.body_token}") + finally: + s.close() + + def test_silent_process_times_out_and_is_killed(self): + with tempfile.TemporaryDirectory() as d: + s = PtySession([sys.executable, "-c", "import time; time.sleep(60)"], Path(d), dict(os.environ), + Path(d) / "t.log", ready_idle=0.3, timeout=2) + with self.assertRaises(TimeoutError): + s.start() + s.close() + self.assertIsNotNone(s.proc.poll()) + + def test_no_reply_line_is_reported(self): + with tempfile.TemporaryDirectory() as d: + repo, s = self._session(d, None) + try: + s.timeout = 3 + res = s.ask("hello") + self.assertEqual(res.reply_text, "") + self.assertIn("no PROBE-REPLY line", res.notes) + finally: + s.close() + + +from harness import ENTRIES # noqa: E402 + +# Adapters that probe a real binary at construction are pinned to one mode here, so the +# self-tests neither spawn a vendor nor depend on which one is installed. +ADAPTER_TEST_ENV = {"KCAP_PROBE_CLAUDE_REAL_CONFIG": "0"} + + +class AdapterHookFilesTests(unittest.TestCase): + def test_every_adapter_writes_a_hook_referencing_the_script(self): + for name, cls in ENTRIES.items(): + if name == "fake": + continue + with self.subTest(entry=name), tempfile.TemporaryDirectory() as d, \ + mock.patch.dict(os.environ, dict(ADAPTER_TEST_ENV, HOME=d)): + a = cls() + sb = new_sandbox(a.lever, None, [], base=Path(d)) + a.prepare(sb) + script = sb.config_root / "probe-hook.sh" + script.write_text("#!/bin/sh\nexit 0\n") + info = a.install_startup_hook(sb, script) + try: + self.assertIsNotNone(info, name) + self.assertTrue(Path(info.config_path).exists(), info) + self.assertIn(str(script), Path(info.config_path).read_text()) + self.assertTrue(info.mechanism) + self.assertIn(a.native_root, a.documented_roots, name) + finally: + # An adapter that must write into the real home removes its hook after a turn. + getattr(a, "cleanup_hook", lambda _sb: None)(sb) + + +class ClaudeAdapterTests(unittest.TestCase): + def test_isolated_mode_trusts_the_repo(self): + from harness.claude import ClaudeAdapter + with tempfile.TemporaryDirectory() as d, mock.patch.dict(os.environ, {"KCAP_PROBE_CLAUDE_REAL_CONFIG": "0"}): + a = ClaudeAdapter() + sb = new_sandbox(a.lever, None, [], base=Path(d)) + a.prepare(sb) + cfg = json.loads((sb.config_root / ".claude.json").read_text()) + self.assertTrue(cfg["projects"][str(sb.repo)]["hasTrustDialogAccepted"]) + argv = a.print_argv(sb, "hi") + self.assertEqual(argv[1:3], ["-p", "hi"]) + self.assertEqual(argv[-2:], ["--setting-sources", "user"]) + self.assertNotIn("KCAP_SKIP", sb.env) + + def test_real_mode_uses_the_real_root_with_settings_flag(self): + from harness.claude import ClaudeAdapter + with tempfile.TemporaryDirectory() as d, mock.patch.dict(os.environ, {"KCAP_PROBE_CLAUDE_REAL_CONFIG": "1"}): + a = ClaudeAdapter() + sb = new_sandbox(a.lever, None, [], base=Path(d)) + a.prepare(sb) + self.assertNotIn("CLAUDE_CONFIG_DIR", sb.env) + self.assertEqual(sb.env["KCAP_SKIP"], "1") + argv = a.print_argv(sb, "hi") + self.assertIn("--settings", argv) + self.assertEqual(argv[argv.index("--settings") + 1], str(sb.config_root / "settings.json")) + self.assertEqual(argv[argv.index("--setting-sources") + 1], "project") + info = a.install_startup_hook(sb, sb.config_root / "probe-hook.sh") + self.assertIn("real config root", info.mechanism) + + +class CodexAdapterTests(unittest.TestCase): + def test_prepare_trusts_repo_and_print_argv(self): + from harness.codex import CodexAdapter + with tempfile.TemporaryDirectory() as d: + a = CodexAdapter() + sb = new_sandbox(a.lever, None, [], base=Path(d)) + a.prepare(sb) + toml = (sb.config_root / "config.toml").read_text() + self.assertIn(f'[projects."{sb.repo}"]', toml) + self.assertIn('trust_level = "trusted"', toml) + info = a.install_startup_hook(sb, sb.config_root / "probe-hook.sh") + hooks = json.loads(Path(info.config_path).read_text()) + self.assertEqual(hooks["hooks"]["SessionStart"][0]["hooks"][0]["command"], str(sb.config_root / "probe-hook.sh")) + + def test_tool_items_distinguish_a_listed_read_from_a_search(self): + from harness.codex import appserver_items, classify_tool_items, exec_items + lines = [ + json.dumps({"type": "item.completed", "item": {"type": "agent_message", "text": "hi"}}), + json.dumps({"type": "item.completed", "item": {"type": "command_execution", + "command": "/bin/zsh -lc 'cat /r/.agents/skills/kcap-probe-1/SKILL.md'"}}), + json.dumps({"type": "item.completed", "item": {"type": "command_execution", + "command": "/bin/zsh -lc 'find / -name SKILL.md'"}}), + json.dumps({"type": "item.completed", "item": {"type": "command_execution", "command": "date"}}), + ] + self.assertEqual(classify_tool_items(exec_items("\n".join(lines))), "tools_used=2 skill_reads=1 searches=1") + frames = [{"frame": {"method": "item/completed", "params": {"item": {"type": "commandExecution", + "command": "cat /r/.agents/skills/x/SKILL.md"}}}}, + {"frame": {"method": "item/completed", "params": {"item": {"type": "agentMessage", "text": "t"}}}}] + self.assertEqual(classify_tool_items(appserver_items(json.dumps(frames))), "tools_used=0 skill_reads=1 searches=0") + + +class CopilotAdapterTests(unittest.TestCase): + def test_skill_tool_is_the_native_load(self): + from harness.copilot import acp_tool_calls, classify_copilot_tools, print_tool_calls + lines = [json.dumps({"type": "tool.execution_start", "data": {"toolName": "skill", "arguments": {"skill": "x"}}}), + json.dumps({"type": "tool.execution_start", "data": {"toolName": "bash", "arguments": {"command": "find"}}}), + json.dumps({"type": "assistant.message", "data": {"content": "hi"}})] + self.assertEqual(classify_copilot_tools(print_tool_calls("\n".join(lines))), + "tools_used=1 skill_loads=1 searches=1") + frames = [{"frame": {"method": "session/update", "params": {"update": { + "sessionUpdate": "tool_call", "title": "Using skill: kcap-probe-1", "kind": "other"}}}}, + {"frame": {"method": "session/update", "params": {"update": { + "sessionUpdate": "tool_call", "title": "Run command", "kind": "execute"}}}}] + self.assertEqual(classify_copilot_tools(acp_tool_calls(json.dumps(frames))), + "tools_used=1 skill_loads=1 searches=0") + + +def _row(entry, scenario, arm, root, verdict, mode, mechanism=None): + return dict(entry=entry, harness=entry, version="1.0", os="o", mode=mode, scenario=scenario, arm=arm, + root=root, exclusion="none", verdict=verdict, flaky=False, runs=2, mechanism=mechanism, + evidence=[], notes="") + + +class ReportTests(unittest.TestCase): + def test_summary_columns(self): + import report + base = dict(entry="x", harness="x", version="1.0", os="o", mode="print", exclusion="none", + flaky=False, runs=2, mechanism=None, evidence=[], notes="") + rows = [ + dict(base, scenario="S1", arm="S1/native", root=".x/skills", verdict="visible_first_turn"), + dict(base, scenario="S2", arm="S2/hook-creates-root", root=".x/skills", verdict="not_visible", mechanism="hook"), + dict(base, scenario="S2", arm="S2/registration", root=".x/skills", verdict="visible_after_reload", mechanism="ext"), + dict(base, scenario="S3", arm="S3/gitignore", root=".x/skills", verdict="visible_first_turn", exclusion="gitignore"), + dict(base, scenario="S3", arm="S3/info-exclude", root=".x/skills", verdict="not_visible", exclusion="info-exclude"), + dict(base, scenario="S4", arm="S4/all-roots", root=".x/skills", verdict="visible_first_turn"), + dict(base, scenario="S4", arm="S4/all-roots", root=".agents/skills", verdict="visible_first_turn"), + dict(base, scenario="S4", arm="S4/all-roots", root=".y/skills", verdict="leaked"), + dict(base, entry="y", harness="y", scenario="S4", arm="S4/all-roots", root=".agents/skills", + verdict="visible_first_turn"), + dict(base, entry="y", harness="y", scenario="S1", arm="S1/native", root=".y/skills", + verdict="untested", notes="binary not installed"), + ] + summary = {s["Entry"]: s for s in report.summarise(rows)} + x = summary["x"] + self.assertEqual(x["Roots consumed"], ".agents/skills, .x/skills (undocumented: .y/skills)") + self.assertEqual(x["Startup mechanism proven"], "ext (print)") + self.assertEqual(x["Exclusion preserving load"], "gitignore") + self.assertEqual(x["Vendor-isolated destination"], ".x/skills") + self.assertEqual(x["Reload path"], "ext") + self.assertEqual(x["_status"], "measured") + self.assertTrue(summary["y"]["_status"].startswith("untested: binary not installed")) + self.assertEqual(summary["y"]["Minimum version"], "—") + text = report.render(list(summary.values())) + self.assertIn("| x | 1.0 |", text) + + def test_two_entries_of_one_vendor_are_one_consumer(self): + rows = [ + _row("v1", "S1", "S1/native", ".v/skills", "visible_first_turn", "print"), + _row("v1", "S4", "S4/all-roots", ".v/skills", "visible_first_turn", "print"), + _row("v2", "S1", "S1/native", ".v/skills", "visible_first_turn", "print"), + _row("v2", "S4", "S4/all-roots", ".v/skills", "visible_first_turn", "print"), + _row("other", "S1", "S1/native", ".o/skills", "visible_first_turn", "print"), + _row("other", "S4", "S4/all-roots", ".v/skills", "leaked", "print"), + ] + for r in rows: + r["harness"] = "theirs" if r["entry"] == "other" else "ours" + import report + s = {r["Entry"]: r for r in report.summarise(rows)} + # Two configurations of one CLI are one consumer; a vendor that leaks into the root is not. + self.assertEqual(s["v1"]["Vendor-isolated destination"], "none") + for r in rows: + if r["entry"] == "other" and r["scenario"] == "S4": + r["root"] = ".o/skills" + s = {r["Entry"]: r for r in report.summarise(rows)} + self.assertEqual(s["v1"]["Vendor-isolated destination"], ".v/skills") + self.assertEqual(s["v2"]["Vendor-isolated destination"], ".v/skills") + + def test_minimum_version_is_the_earliest_proven_one(self): + rows = [ + _row("x", "S1", "S1/native", ".x/skills", "visible_first_turn", "print"), + _row("x", "S1", "S1/native", ".x/skills", "visible_first_turn", "print"), + _row("x", "S1", "S1/native", ".x/skills", "untested", "print"), + ] + rows[0]["version"], rows[1]["version"], rows[2]["version"] = "1.10.0", "1.9.0", "0.1.0" + import report + s = {r["Entry"]: r for r in report.summarise(rows)}["x"] + # Ordered by number, not by text, and only versions the control actually passed on. + self.assertEqual(s["Minimum version"], "1.9.0") + + def test_lifecycle_columns(self): + rows = [ + _row("x", "S1", "S1/native", ".x/skills", "visible_first_turn", "print"), + _row("x", "S5", "S5/add", ".x/skills", "visible_live", "daemon"), + _row("x", "S5", "S5/delete", ".x/skills", "revoked", "daemon"), + _row("x", "S6", "S6/update", ".x/skills", "stale", "print"), + _row("x", "S6", "S6/update", ".x/skills", "stale", "daemon"), + _row("x", "S7", "S7/add", ".x/skills", "untested", "print"), + _row("x", "S9", "S9/linked-other", ".x/skills", "not_visible", "print"), + _row("x", "S1", "S1/native", ".x/skills", "visible_first_turn", "tui"), + _row("x", "S5", "S5/reload", ".x/skills", "visible_after_reload", "tui", mechanism="/reload"), + _row("y", "S1", "S1/native", ".y/skills", "visible_first_turn", "print"), + _row("y", "S5", "S5/add", ".y/skills", "visible_live", "tui"), + ] + import report + by_entry = {r["Entry"]: r for r in report.summarise(rows)} + # S5 ran only interactively: the daemon-mode column must not read as a measured "none". + self.assertEqual(by_entry["y"]["Live catalogue"], "n/a (not run)") + self.assertEqual(by_entry["y"]["Interactive"], "add=visible_live") + s = by_entry["x"] + self.assertEqual(s["Live catalogue"], "add=visible_live; delete=revoked") + self.assertEqual(s["Startup rewrite"], "update=stale") + self.assertEqual(s["Resume"], "n/a (not run)") + self.assertEqual(s["Nested cwd"], "n/a (not run)") + self.assertEqual(s["Worktree"], "linked-other=not_visible") + self.assertEqual(s["Interactive"], "S1=visible_first_turn; reload=visible_after_reload") + self.assertIn("/reload", s["Reload path"]) + self.assertIn("tui", s["Modes"]) + + +class CursorAdapterTests(unittest.TestCase): + def test_acp_read_of_the_listed_file_is_the_native_load(self): + from harness.cursor import classify_cursor_tools + def upd(**u): + return {"frame": {"method": "session/update", "params": {"update": u}}} + frames = [ + upd(sessionUpdate="tool_call", toolCallId="a", kind="read", title="Read File"), + upd(sessionUpdate="tool_call_update", toolCallId="a", rawInput={"path": "/r/.cursor/skills/x/SKILL.md"}), + upd(sessionUpdate="tool_call", toolCallId="b", kind="execute", title="Run command", rawInput={"command": "find"}), + upd(sessionUpdate="agent_message_chunk", content={"type": "text", "text": "hi"}), + ] + self.assertEqual(classify_cursor_tools(json.dumps(frames)), "tools_used=1 skill_reads=1 searches=1") + + def test_print_stream_read_of_the_listed_file_is_the_native_load(self): + from harness.cursor import classify_cursor_stream + lines = [ + json.dumps({"type": "tool_call", "subtype": "started", "call_id": "a", + "tool_call": {"readToolCall": {"args": {"path": "/r/.cursor/skills/x/SKILL.md"}}}}), + json.dumps({"type": "tool_call", "subtype": "completed", "call_id": "a", + "tool_call": {"readToolCall": {"args": {"path": "/r/.cursor/skills/x/SKILL.md"}}}}), + json.dumps({"type": "tool_call", "subtype": "started", "call_id": "b", + "tool_call": {"shellToolCall": {"args": {"command": "find / -name SKILL.md"}}}}), + json.dumps({"type": "assistant", "message": {"content": [{"type": "text", "text": "hi"}]}}), + ] + self.assertEqual(classify_cursor_stream("\n".join(lines)), "tools_used=1 skill_reads=1 searches=1") + + def test_user_hooks_variant_restores_the_real_file(self): + from harness.cursor import CursorUserHooksAdapter + with tempfile.TemporaryDirectory() as d, mock.patch.dict(os.environ, {"HOME": d}): + hooks = Path(d) / ".cursor" / "hooks.json" + hooks.parent.mkdir(parents=True) + original = json.dumps({"version": 1, "hooks": {"sessionStart": [{"command": "kcap hook --cursor"}]}}) + hooks.write_text(original) + a = CursorUserHooksAdapter() + sb = new_sandbox(a.lever, None, [], base=Path(d)) + info = a.install_startup_hook(sb, sb.config_root / "probe-hook.sh") + merged = json.loads(Path(info.config_path).read_text()) + self.assertEqual([h["command"] for h in merged["hooks"]["sessionStart"]], + ["kcap hook --cursor", str(sb.config_root / "probe-hook.sh")]) + a.cleanup_hook(sb) + self.assertEqual(hooks.read_text(), original) + + +class CursorRegistrationTests(unittest.TestCase): + def test_registration_names_the_path_it_writes(self): + from harness.cursor import CursorAdapter, CursorUserHooksAdapter + for cls in (CursorAdapter, CursorUserHooksAdapter): + with self.subTest(entry=cls.entry), tempfile.TemporaryDirectory() as d: + a = cls() + sb = new_sandbox(a.lever, None, [], base=Path(d)) + try: + skill = ProbeSkill.fresh() + arm_target = a.skill_file(sb, a.native_root, skill.name) + info = a.install_registration(sb, arm_target, skill.render()) + # The hook writes into a plugin directory, not the arm's own path, and the arm + # checks the path the mechanism names. + self.assertIsNotNone(info.target) + self.assertNotEqual(info.target, arm_target) + subprocess.run([json.loads(Path(info.config_path).read_text())["hooks"]["workspaceOpen"][0]["command"]], + capture_output=True, text=True, timeout=20) + self.assertTrue(info.target.exists()) + self.assertIn(skill.body_token, info.target.read_text()) + finally: + getattr(a, "cleanup_hook", lambda _sb: None)(sb) + sb.cleanup() + + +class KiroAdapterTests(unittest.TestCase): + def test_acp_read_of_the_listed_file_is_the_native_load(self): + from harness.kiro import classify_kiro_tools + def upd(**u): + return {"frame": {"method": "session/update", "params": {"update": u}}} + frames = [ + upd(sessionUpdate="tool_call", toolCallId="a", kind="read", title="Reading SKILL.md:1", + locations=[{"path": "/r/.kiro/skills/x/SKILL.md"}]), + upd(sessionUpdate="tool_call_update", toolCallId="a", status="completed"), + upd(sessionUpdate="tool_call", toolCallId="b", kind="execute", title="find / -name SKILL.md"), + ] + self.assertEqual(classify_kiro_tools(json.dumps(frames)), "tools_used=1 skill_reads=1 searches=1") + + def test_version_picks_the_hook_generation(self): + from harness.kiro import KiroAdapter + with tempfile.TemporaryDirectory() as d: + a = KiroAdapter() + a.version = lambda env=None: "kiro-cli 2.21.4" + sb = new_sandbox(a.lever, None, [], base=Path(d)) + info = a.install_startup_hook(sb, sb.config_root / "probe-hook.sh") + self.assertIn("agentSpawn", info.mechanism) + self.assertEqual(json.loads((sb.config_root / "settings" / "cli.json").read_text())["chat.defaultAgent"], "probe") + a.version = lambda env=None: "kiro-cli 3.0.1" + info = a.install_startup_hook(sb, sb.config_root / "probe-hook.sh") + self.assertIn("SessionStart", info.mechanism) + self.assertEqual(json.loads(Path(info.config_path).read_text())["hooks"][0]["trigger"], "SessionStart") + + +class PiAdapterTests(unittest.TestCase): + def test_registration_extension_names_the_root(self): + from harness.pi import PiAdapter + with tempfile.TemporaryDirectory() as d: + a = PiAdapter() + sb = new_sandbox(a.lever, None, [], base=Path(d)) + skill = ProbeSkill.fresh() + target = a.skill_file(sb, a.native_root, skill.name) + info = a.install_registration(sb, target, skill.render()) + ext = Path(info.config_path).read_text() + self.assertIn("resources_discover", ext) + self.assertIn(json.dumps(str(sb.repo / ".pi" / "skills")), ext) + self.assertIn(json.dumps(str(sb.config_root / "probe-hook.sh")), ext) + self.assertTrue((sb.config_root / "probe-hook.sh").exists()) + + +import importlib # noqa: E402 +import re # noqa: E402 + +from harness import ENTRIES # noqa: E402 + +# (entry, the module holding its print driver, the flags a resumed launch must carry) +RESUME_CASES = ( + ("claude", "harness.claude", ["--resume", "sid-1"]), + ("codex", "harness.codex", ["resume", "sid-1"]), + ("copilot", "harness.copilot", ["--resume=sid-1"]), + ("pi", "harness.pi", ["--session", "sid-1"]), + ("kiro", "harness.kiro", ["--resume"]), + ("cursor", "harness.cursor", ["--resume=sid-1"]), + ("opencode-v1", "harness.opencode_v1", ["--session", "sid-1"]), + ("agy", "harness.agy", ["--conversation", "sid-1"]), + ("gemini", "harness.gemini", ["--resume", "sid-1"]), +) + +SESSION_ID_CASES = ( + ("claude", '{"session_id": "cl-1", "result": "hi"}', "cl-1"), + ("codex", '{"type": "thread.started", "thread_id": "cx-1"}\n{"type": "item.completed"}', "cx-1"), + ("copilot", '{"type": "assistant.message"}\n{"type": "result", "sessionId": "co-1"}', "co-1"), + ("pi", '{"type": "session", "id": "pi-1"}\n{"type": "agent_start"}', "pi-1"), + ("cursor", '{"type": "system", "session_id": "cu-1"}', "cu-1"), + ("opencode-v1", '{"part": {"sessionID": "oc-1", "type": "text"}}', "oc-1"), + ("agy", '{"event": "step_update", "conversation_id": "ag-1"}', "ag-1"), +) + + +class AdapterPassTwoTests(unittest.TestCase): + + def _stub(self, d: str): + sb = new_sandbox("PROBE_UNUSED_LEVER", None, [], base=Path(d)) + self.addCleanup(sb.cleanup) + return sb + + def test_resume_launch_carries_the_session(self): + for entry, module_name, expected in RESUME_CASES: + with self.subTest(entry=entry), tempfile.TemporaryDirectory() as d: + module = importlib.import_module(module_name) + a = ENTRIES[entry]() + sb = self._stub(d) + a.prepare(sb) + seen: list[tuple[list[str], str]] = [] + + def fake_print_ask(argv, cwd, env, stderr_path, timeout, stdin_text=None, extract=None): + seen.append((list(argv), stdin_text or "")) + return AskResult(reply_text="", raw="", argv=list(argv), started_at=0.0, + first_request_at=0.0, stderr_path=None, exit_code=0) + + with mock.patch.object(module, "print_ask", fake_print_ask): + a.resume(sb, "sid-1", "prompt") + argv, stdin_text = seen[0] + for flag in expected: + self.assertIn(flag, argv) + # Codex reads its prompt from stdin; the others carry it in the command line. + self.assertIn("prompt", " ".join(argv) + stdin_text) + + def test_session_id_is_read_from_the_vendor_stream(self): + for entry, raw, expected in SESSION_ID_CASES: + with self.subTest(entry=entry): + a = ENTRIES[entry]() + res = AskResult(reply_text="", raw=raw, argv=[], started_at=0.0, first_request_at=0.0, + stderr_path=None, exit_code=0) + self.assertEqual(a.session_id(res), expected) + blank = AskResult(reply_text="", raw="not json", argv=[], started_at=0.0, + first_request_at=0.0, stderr_path=None, exit_code=0) + self.assertIsNone(a.session_id(blank)) + + def test_kiro_resumes_the_launch_directory_without_an_id(self): + a = ENTRIES["kiro"]() + res = AskResult(reply_text="ok", raw="", argv=[], started_at=0.0, first_request_at=0.0, + stderr_path=None, exit_code=0) + # Its non-interactive output carries no id, so the arm must still have something to resume. + self.assertTrue(a.session_id(res)) + + def test_every_declared_mode_has_a_launch(self): + with tempfile.TemporaryDirectory() as d: + sb = self._stub(d) + for entry, cls in ENTRIES.items(): + with self.subTest(entry=entry): + a = cls() + if "tui" in a.modes: + argv = a.tui_argv(sb) + self.assertTrue(argv, f"{entry} declares tui with no argv") + self.assertNotIn("-p", argv) + self.assertNotIn("--print", argv) + if a.can_resume: + self.assertIsNot(type(a).resume, Adapter.resume, f"{entry} cannot resume") + self.assertIsNot(type(a).session_id, Adapter.session_id, f"{entry} reads no id") + + def test_dialog_answers_are_one_key_at_a_time(self): + from lib.pty_driver import KEY_RE + for entry, cls in ENTRIES.items(): + for pattern, keys in cls.tui_dialogs: + with self.subTest(entry=entry, dialog=pattern): + self.assertTrue(re.compile(pattern)) + # An arrow and its Enter must survive as two keystrokes, not one write. + self.assertEqual("".join(KEY_RE.findall(keys)), keys) + + +if __name__ == "__main__": + unittest.main() diff --git a/docs/probes/2026-09-16-skills-discovery/selftest_servers.py b/docs/probes/2026-09-16-skills-discovery/selftest_servers.py new file mode 100755 index 0000000000..f7fdf61f71 --- /dev/null +++ b/docs/probes/2026-09-16-skills-discovery/selftest_servers.py @@ -0,0 +1,151 @@ +#!/usr/bin/env python3 +"""Fake vendor processes for the kit's self-tests: acp, appserver (codex app-server) and pirpc.""" +from __future__ import annotations + +import json +import os +import subprocess +import re +import sys +from pathlib import Path + +TOKEN_RE = re.compile(r"PROBE-BODY-([0-9a-f]{12})") +UNTRUSTED_HOOK_KEY = "/x/hooks.json:SessionStart:0:0" +UNTRUSTED_HOOK_HASH = "sha256:fake" + + +def answer() -> str: + lines = [] + for f in sorted(Path.cwd().glob(".fake/skills/kcap-probe-*/SKILL.md")): + m = TOKEN_RE.search(f.read_text()) + if m: + lines.append(f"{f.parent.name}=PROBE-BODY-{m.group(1)}") + return "\n".join(lines) if lines else "NO-SKILL" + + +def send(obj: dict) -> None: + sys.stdout.write(json.dumps(obj) + "\n") + sys.stdout.flush() + + +def uses_tool() -> bool: + return os.environ.get("KCAP_FAKE_TOOL_CALL") == "1" + + +def acp() -> None: + if os.environ.get("KCAP_FAKE_GRANDCHILD") == "1": + subprocess.Popen(["sleep", "60"]) + for line in sys.stdin: + msg = json.loads(line) + m, i, p = msg.get("method"), msg.get("id"), msg.get("params") or {} + if m == "initialize": + send({"jsonrpc": "2.0", "id": i, "result": {"protocolVersion": 1, "agentCapabilities": {}}}) + elif m == "session/new": + send({"jsonrpc": "2.0", "id": i, "result": {"sessionId": "s1"}}) + elif m == "session/prompt": + if uses_tool(): + send({"jsonrpc": "2.0", "method": "session/update", "params": { + "sessionId": p["sessionId"], + "update": {"sessionUpdate": "tool_call", "toolCallId": "t1", "title": "read", + "status": "completed"}}}) + send({"jsonrpc": "2.0", "method": "session/update", "params": { + "sessionId": p["sessionId"], + "update": {"sessionUpdate": "agent_message_chunk", "content": {"type": "text", "text": answer()}}}}) + send({"jsonrpc": "2.0", "id": i, "result": {"stopReason": "end_turn"}}) + elif i is not None: + send({"jsonrpc": "2.0", "id": i, "error": {"code": -32601, "message": "unknown"}}) + + +def hooks_list() -> list[dict]: + if os.environ.get("KCAP_FAKE_UNTRUSTED_HOOK") != "1": + return [] + seeded = any(a.startswith("hooks.state=") for a in sys.argv) + return [{"key": UNTRUSTED_HOOK_KEY, "trustStatus": "trusted" if seeded else "untrusted", + "currentHash": UNTRUSTED_HOOK_HASH}] + + +def appserver() -> None: + for line in sys.stdin: + msg = json.loads(line) + m, i, p = msg.get("method"), msg.get("id"), msg.get("params") or {} + if m == "initialize": + send({"jsonrpc": "2.0", "id": i, "result": {}}) + elif m == "hooks/list": + # The real app-server groups hooks per working directory under `data`. + send({"jsonrpc": "2.0", "id": i, "result": {"data": [ + {"cwd": str(Path.cwd()), "hooks": hooks_list(), "warnings": [], "errors": []}]}}) + elif m == "thread/start": + send({"jsonrpc": "2.0", "id": i, "result": {"thread": {"id": "t1"}, "model": "fake"}}) + elif m == "turn/start": + send({"jsonrpc": "2.0", "id": i, "result": {"turn": {"id": "u1"}}}) + if uses_tool(): + send({"jsonrpc": "2.0", "method": "item/completed", "params": { + "item": {"type": "commandExecution", "id": "c1", "command": "cat SKILL.md"}}}) + send({"jsonrpc": "2.0", "method": "item/completed", "params": { + "item": {"type": "agentMessage", "id": "m1", "text": answer()}}}) + send({"jsonrpc": "2.0", "method": "turn/completed", "params": {"turn": {"id": "u1", "status": "completed"}}}) + elif i is not None: + send({"jsonrpc": "2.0", "id": i, "error": {"code": -32601, "message": "unknown"}}) + + +def pirpc() -> None: + if os.environ.get("KCAP_FAKE_GRANDCHILD") == "1": + # A vendor that re-execs leaves a process like this one holding stdout after it exits. + subprocess.Popen(["sleep", "60"]) + for line in sys.stdin: + msg = json.loads(line) + if msg.get("type") == "prompt": + send({"id": msg.get("id"), "type": "response", "success": True}) + send({"type": "agent_start"}) + content = [{"type": "text", "text": answer()}] + if uses_tool(): + # A real Pi tool call shows up twice: as a toolCall part and as a tool_execution_end. + content.insert(0, {"type": "toolCall", "id": "t1", "name": "read", "arguments": {}}) + send({"type": "tool_execution_end", "toolCallId": "t1", "status": "success"}) + send({"type": "message_end", "message": {"role": "assistant", "content": content}}) + send({"type": "agent_settled"}) + + +def tui() -> None: + hook = os.environ.get("KCAP_FAKE_HOOK") + if hook: + subprocess.run([hook], input="{}", capture_output=True, text=True, timeout=10) + frozen = answer() if os.environ.get("KCAP_FAKE_TUI_FROZEN") == "1" else None + out = sys.stdout + out.write("\x1b[1mfake tui\x1b[0m ready\r\n") + if os.environ.get("KCAP_FAKE_TUI_DIALOG") == "1": + out.write("Do you trust this folder? (y/n) ") + out.flush() + if not sys.stdin.readline().strip().lower().startswith("y"): + return + out.write("> ") + out.flush() + for line in sys.stdin: + line = line.strip() + if line in ("/exit", "/quit"): + return + if line == "/reload": + if frozen is not None: + frozen = answer() + out.write("reloaded\r\n> ") + elif line.startswith("You have a skill"): + reply = frozen if frozen is not None else answer() + entries = [] if reply == "NO-SKILL" else reply.splitlines() + # The prompt embeds the queried skill's name; a reply for some other skill still on + # disk must not be mistaken for it. + match = next((e for e in entries if e.split("=", 1)[0] in line), None) + value = match.split("=", 1)[1] if match else "NO-SKILL" + mark = re.search(r"PROBE-MARK-[0-9a-f]{4}", line) + if "PROBE-REPLY" in line and os.environ.get("KCAP_FAKE_TUI_BARE") != "1": + marked = f"{mark.group(0)} " if mark else "" + # A spinner redrawing the status line while the answer streams beside it. + out.write(f"\x1b[32m**PROBE-REPLY: {marked}{value}**\x1b[0m\r\n\x1b[A\x1b[999C working\r\n> ") + else: + out.write(f"{value}\r\n> ") + else: + out.write("?\r\n> ") + out.flush() + + +if __name__ == "__main__": + {"acp": acp, "appserver": appserver, "app-server": appserver, "pirpc": pirpc, "tui": tui}[sys.argv[1]]() diff --git a/docs/superpowers/plans/2026-09-16-ai2829-skills-discovery-probes.md b/docs/superpowers/plans/2026-09-16-ai2829-skills-discovery-probes.md new file mode 100644 index 0000000000..acc888372e --- /dev/null +++ b/docs/superpowers/plans/2026-09-16-ai2829-skills-discovery-probes.md @@ -0,0 +1,3627 @@ +# Skills Discovery Probes Implementation Plan (pass 1) + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** A reproducible Python probe kit under `docs/probes/2026-09-16-skills-discovery/` that measures, per harness and launch mode, whether a repo-local skill written by a startup hook is visible to the first model request, which Git exclusion keeps it loadable, and which vendor directories each harness consumes, and emits `matrix.json`, `findings.md` and `capability-matrix.md`. + +**Architecture:** A shared library (`lib/`) owns the sandbox (fresh git repo + private vendor config root), the nonce probe skill and its prompts, Git exclusion, the run recorder and verdict rules. One adapter per harness (`harness/`) implements four primitives: install the vendor's startup hook into the isolated user-level config, launch in a mode, send the first prompt and return the reply, optionally list the catalogue. `probe.py` runs scenarios S0 to S4 over adapters and rebuilds the matrix from run files. Vendor hook configuration is hand-written by the adapters so a verdict describes the vendor, never kcap. + +**Tech Stack:** Python 3.14 standard library only (no third-party packages); `git`; the vendor CLIs installed on the developer machine; the `AcpClient` from `docs/probes/2026-08-04-acp-reconnect-c0/acp_c0_probe.py` for ACP daemon modes. + +**Spec:** `docs/superpowers/specs/2026-09-16-skills-discovery-probes-design.md` + +## Global Constraints + +- No production code changes: nothing under `src/` or `test/` is touched. Kit files live only under `docs/probes/2026-09-16-skills-discovery/`. +- Standard library only. `python3` on the machine is 3.14.5; use `from __future__ import annotations` and nothing newer than 3.11 syntax so an older machine can run the kit. +- Every path handed to a vendor is resolved with `Path.resolve()` (macOS `/var` is a symlink to `/private`). +- Vendor processes get a minimal allow-listed environment plus the vendor's config-root lever; nothing from the developer's `.envrc` leaks in. +- Credential files are copied by explicit per-adapter lists, never whole directories. +- `out/` and `*.stderr.log` are git-ignored; `matrix.json`, `findings.md`, `capability-matrix.md` are committed. +- Comments are scarce (see CLAUDE.md "Comments"): no history, no design coordinates, no Linear ids anywhere in kit code. Docs may cite `#961`. +- Commit subjects: one imperative clause, at most 80 characters including the trailing `(#961)`, trailer `Co-Authored-By: Claude Fable 5.1 `. +- In this worktree run git as `/usr/bin/git -C /Users/alexey/dev/eventstore/kcap-cli/.claude/worktrees/peppy-percolating-biscuit `, one command per invocation, no heredocs. `KIT` below means `/Users/alexey/dev/eventstore/kcap-cli/.claude/worktrees/peppy-percolating-biscuit/docs/probes/2026-09-16-skills-discovery`. +- Self-tests run as a script: `python3 $KIT/selftest.py -v`. They must pass with no vendor binary installed. +- Each turn arm costs one real model request on the developer's account. Never loop a turn arm beyond the repetition rule (2 runs, a 3rd on disagreement). + +--- + +## File structure + +``` +docs/probes/2026-09-16-skills-discovery/ + .gitignore out/, *.stderr.log + probe.py CLI orchestrator: scenarios, repetition, --emit + selftest.py unittest suite over lib/, harness/base.py and probe.py with a fake adapter + lib/__init__.py + lib/isolation.py Sandbox, new_sandbox(), git() + lib/probe_skill.py ProbeSkill, write_skill(), single_prompt(), multi_prompt(), parse_reply() + lib/git_exclusion.py apply(), assert_untracked_state() + lib/verdict.py judge_single(), judge_root(), combine(), PromptDesignFailure + lib/recorder.py RunRecord, write_run(), load_runs(), emit_matrix() + lib/hook_script.py write_hook_script(), read_stamp() + lib/acp_driver.py acp_ask(): one ACP turn over AcpClient + lib/print_driver.py print_ask(): one print-mode turn over subprocess + harness/__init__.py ENTRIES registry + harness/base.py Adapter, AskResult, HookInfo + harness/fake.py FakeAdapter used by selftest.py + harness/claude.py … harness/agy.py one adapter per entry (Tasks 10-19) + findings.md written in Task 21 + capability-matrix.md written in Task 21 + matrix.json emitted in Task 20 +``` + +Shared vocabulary used by every task: + +- `entry`: `claude`, `codex`, `gemini`, `pi`, `cursor`, `copilot`, `kiro`, `opencode-v1`, `opencode-v2`, `agy`. +- `mode`: `print` or `daemon` (pass 1), `interactive` reserved. +- `scenario`: `S0`, `S1`, `S2`, `S3`, `S4`; `arm` names within a scenario: `S0/none`, `S1/native`, `S2/hook-creates-root`, `S2/hook-adds-skill`, `S2/registration`, `S3/gitignore`, `S3/info-exclude`, `S4/all-roots`, `S4/confirm-`. +- `root`: a repo-relative skills directory such as `.claude/skills`; `root key` is the same string with `/` replaced by `_` for file names. + +--- + +### Task 1: Kit skeleton and self-test runner + +**Files:** +- Create: `docs/probes/2026-09-16-skills-discovery/.gitignore` +- Create: `docs/probes/2026-09-16-skills-discovery/lib/__init__.py` +- Create: `docs/probes/2026-09-16-skills-discovery/harness/__init__.py` +- Create: `docs/probes/2026-09-16-skills-discovery/selftest.py` + +**Interfaces:** +- Produces: `selftest.py` runnable as `python3 $KIT/selftest.py -v`; it puts the kit directory first on `sys.path` so `import lib.x` and `import harness.x` work from any cwd. + +- [ ] **Step 1: Create the directories and the ignore file** + +`.gitignore`: + +``` +out/ +*.stderr.log +``` + +`lib/__init__.py` and `harness/__init__.py` are empty files. + +- [ ] **Step 2: Write the self-test runner** + +`selftest.py`: + +```python +#!/usr/bin/env python3 +"""Self-tests for the skills discovery probe kit. Needs git and Python; no vendor binary.""" +from __future__ import annotations + +import sys +import unittest +from pathlib import Path + +KIT = Path(__file__).resolve().parent +sys.path.insert(0, str(KIT)) + + +class KitLayoutTests(unittest.TestCase): + def test_out_is_ignored(self): + ignore = (KIT / ".gitignore").read_text().splitlines() + self.assertIn("out/", ignore) + self.assertIn("*.stderr.log", ignore) + + +if __name__ == "__main__": + unittest.main() +``` + +- [ ] **Step 3: Run it** + +Run: `python3 $KIT/selftest.py -v` +Expected: `test_out_is_ignored ... ok`, `OK` + +- [ ] **Step 4: Commit** + +``` +/usr/bin/git -C add docs/probes/2026-09-16-skills-discovery +/usr/bin/git -C commit -m "Scaffold the skills discovery probe kit (#961)" -m "Co-Authored-By: Claude Fable 5.1 " +``` + +--- + +### Task 2: Probe skill, prompts and reply parser + +**Files:** +- Create: `docs/probes/2026-09-16-skills-discovery/lib/probe_skill.py` +- Modify: `docs/probes/2026-09-16-skills-discovery/selftest.py` + +**Interfaces:** +- Produces: + - `class ProbeSkill(nonce: str, token: str)` frozen dataclass; `ProbeSkill.fresh() -> ProbeSkill`; properties `name -> str` (`kcap-probe-`), `body_token -> str` (`PROBE-BODY-`), `description -> str`; method `render() -> str` (the SKILL.md text). + - `write_skill(root: Path, skill: ProbeSkill, flat: bool = False) -> Path` writes `//SKILL.md` (or `/.md` when `flat`) and returns the file path. + - `single_prompt(skill: ProbeSkill) -> str`, `multi_prompt() -> str`. + - `class Reply(tokens: frozenset[str], skill_named: bool, no_skill: bool)`; `parse_reply(reply_text: str, raw: str) -> Reply`. `tokens` are the 12-hex values found anywhere in `raw`; `skill_named` is whether any `kcap-probe-<6hex>` appears in `reply_text`; `no_skill` is whether `reply_text` contains `NO-SKILL`. + - Constants `TOKEN_RE`, `NAME_RE`, `NO_SKILL = "NO-SKILL"`. + +- [ ] **Step 1: Write the failing tests** + +Append to `selftest.py` (above `if __name__`): + +```python +from lib.probe_skill import ( # noqa: E402 + NO_SKILL, ProbeSkill, multi_prompt, parse_reply, single_prompt, write_skill, +) + + +class ProbeSkillTests(unittest.TestCase): + def test_token_only_in_body(self): + s = ProbeSkill.fresh() + text = s.render() + head, _, body = text.partition("\n---\n") + self.assertIn(f"name: {s.name}", head) + self.assertNotIn(s.token, head) + self.assertIn(s.body_token, body) + self.assertNotIn(s.token, single_prompt(s)) + self.assertNotIn(s.token, multi_prompt()) + self.assertIn(s.name, single_prompt(s)) + + def test_write_skill_layouts(self): + import tempfile + s = ProbeSkill.fresh() + with tempfile.TemporaryDirectory() as d: + root = Path(d) / ".claude" / "skills" + f = write_skill(root, s) + self.assertEqual(f, root / s.name / "SKILL.md") + self.assertEqual(f.read_text(), s.render()) + flat = write_skill(Path(d) / "flat", s, flat=True) + self.assertEqual(flat.name, f"{s.name}.md") + + def test_parse_reply(self): + s = ProbeSkill.fresh() + r = parse_reply(f"The token is {s.body_token}.", f'{{"result":"The token is {s.body_token}."}}') + self.assertEqual(r.tokens, frozenset({s.token})) + self.assertFalse(r.skill_named) + r = parse_reply(f"I can see {s.name} but cannot read it", "") + self.assertEqual(r.tokens, frozenset()) + self.assertTrue(r.skill_named) + r = parse_reply(NO_SKILL, NO_SKILL) + self.assertTrue(r.no_skill) + # raw is the fallback only when no reply text was extracted + r = parse_reply("", f"chunk: {s.body_token}") + self.assertEqual(r.tokens, frozenset({s.token})) + r = parse_reply(NO_SKILL, f'{{"tool_result":"{s.body_token}"}}') + self.assertEqual(r.tokens, frozenset()) +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `python3 $KIT/selftest.py -v` +Expected: `ModuleNotFoundError: No module named 'lib.probe_skill'` + +- [ ] **Step 3: Implement** + +`lib/probe_skill.py`: + +```python +from __future__ import annotations + +import re +import secrets +from dataclasses import dataclass +from pathlib import Path + +TOKEN_RE = re.compile(r"PROBE-BODY-([0-9a-f]{12})") +NAME_RE = re.compile(r"kcap-probe-[0-9a-f]{6}") +NO_SKILL = "NO-SKILL" + + +@dataclass(frozen=True) +class ProbeSkill: + nonce: str + token: str + + @staticmethod + def fresh() -> "ProbeSkill": + return ProbeSkill(secrets.token_hex(3), secrets.token_hex(6)) + + @property + def name(self) -> str: + return f"kcap-probe-{self.nonce}" + + @property + def body_token(self) -> str: + return f"PROBE-BODY-{self.token}" + + @property + def description(self) -> str: + return "Answers a question about the probe token for this repository." + + def render(self) -> str: + return ( + "---\n" + f"name: {self.name}\n" + f"description: {self.description}\n" + "---\n" + "When asked for the probe token, reply with exactly this token and nothing else:\n" + f"{self.body_token}\n" + ) + + +def write_skill(root: Path, skill: ProbeSkill, flat: bool = False) -> Path: + if flat: + root.mkdir(parents=True, exist_ok=True) + path = root / f"{skill.name}.md" + else: + (root / skill.name).mkdir(parents=True, exist_ok=True) + path = root / skill.name / "SKILL.md" + path.write_text(skill.render()) + return path + + +NO_TOOLS = ( + "Do not run any tool, search the file system or read any file to find it; only a skill that " + "is already available to you counts." +) + + +def single_prompt(skill: ProbeSkill) -> str: + return ( + f"You have a skill named {skill.name}. Use it and reply with only the probe token it " + f"contains. {NO_TOOLS} If no such skill is available to you, reply with exactly {NO_SKILL}." + ) + + +def multi_prompt() -> str: + return ( + "List every skill available to you whose name starts with kcap-probe-. For each, reply " + "with = on its own line, reading the token from the skill body. " + f"{NO_TOOLS} If there are none, reply with exactly {NO_SKILL}." + ) + + +@dataclass(frozen=True) +class Reply: + tokens: frozenset[str] + skill_named: bool + no_skill: bool + + +def parse_reply(reply_text: str, raw: str) -> Reply: + # The reply decides; the raw event log is consulted only when no reply text was extracted, + # because a tool-call frame that echoes SKILL.md would otherwise count as a loaded skill. + found = TOKEN_RE.findall(reply_text) if reply_text.strip() else TOKEN_RE.findall(raw) + return Reply( + tokens=frozenset(found), + skill_named=NAME_RE.search(reply_text) is not None, + no_skill=NO_SKILL in reply_text, + ) +``` + +- [ ] **Step 4: Run to verify pass** + +Run: `python3 $KIT/selftest.py -v` +Expected: 4 tests, `OK` + +- [ ] **Step 5: Commit** + +``` +/usr/bin/git -C add docs/probes/2026-09-16-skills-discovery +/usr/bin/git -C commit -m "Add the probe skill, prompts and reply parser (#961)" -m "Co-Authored-By: Claude Fable 5.1 " +``` + +--- + +### Task 3: Sandbox isolation + +**Files:** +- Create: `docs/probes/2026-09-16-skills-discovery/lib/isolation.py` +- Modify: `docs/probes/2026-09-16-skills-discovery/selftest.py` + +**Interfaces:** +- Produces: + - `git(repo: Path, *args: str, env: dict | None = None) -> str` runs `git -C `, raises `subprocess.CalledProcessError` on failure, returns stdout. + - `class Sandbox(root: Path, repo: Path, config_root: Path, env: dict[str, str])` with `cleanup()`. + - `new_sandbox(lever: str, real_root: Path | None, credential_files: list[str], passthrough_env: list[str] = (), extra_env: dict[str, str] | None = None, keep: bool = False, base: Path | None = None) -> Sandbox`. `lever` is the env var name that relocates the vendor root (`HOME` for a full home override). `env[lever]` is `str(config_root)`. With `lever != "HOME"`, `env["HOME"]` is the real home. `credential_files` are paths relative to `real_root`, copied to the same relative path under `config_root` when present. `passthrough_env` names extra variables copied from `os.environ` when set. + - `ENV_ALLOWLIST = ("PATH", "TERM", "LANG", "LC_ALL", "LC_CTYPE", "TMPDIR", "SHELL", "USER", "LOGNAME")`. + +- [ ] **Step 1: Write the failing tests** + +Append to `selftest.py`: + +```python +import os # noqa: E402 +import tempfile # noqa: E402 +from unittest import mock # noqa: E402 + +from lib.isolation import ENV_ALLOWLIST, Sandbox, git, new_sandbox # noqa: E402 + + +class IsolationTests(unittest.TestCase): + def test_sandbox_repo_and_lever(self): + with tempfile.TemporaryDirectory() as d: + real = Path(d) / "real" + (real / "nested").mkdir(parents=True) + (real / "auth.json").write_text("{}") + (real / "nested" / "creds").write_text("x") + (real / "secret.db").write_text("no") + with mock.patch.dict(os.environ, {"KCAP_URL": "https://leak.invalid"}): + sb = new_sandbox("CODEX_HOME", real, ["auth.json", "nested/creds", "missing.json"], + base=Path(d)) + try: + self.assertEqual(sb.repo, sb.repo.resolve()) + self.assertEqual(git(sb.repo, "rev-list", "--count", "HEAD").strip(), "1") + self.assertEqual(sb.env["CODEX_HOME"], str(sb.config_root)) + self.assertEqual(sb.env["HOME"], os.environ["HOME"]) + self.assertNotIn("KCAP_URL", sb.env) + self.assertTrue((sb.config_root / "auth.json").exists()) + self.assertTrue((sb.config_root / "nested" / "creds").exists()) + self.assertFalse((sb.config_root / "secret.db").exists()) + for k in sb.env: + self.assertTrue(k in ENV_ALLOWLIST or k in ("HOME", "CODEX_HOME"), k) + finally: + sb.cleanup() + self.assertFalse(sb.root.exists()) + + def test_home_lever_and_passthrough(self): + with tempfile.TemporaryDirectory() as d: + with mock.patch.dict(os.environ, {"GH_TOKEN": "t", "OTHER": "o"}): + sb = new_sandbox("HOME", None, [], passthrough_env=["GH_TOKEN"], + extra_env={"XDG_CONFIG_HOME": "/x"}, base=Path(d)) + try: + self.assertEqual(sb.env["HOME"], str(sb.config_root)) + self.assertEqual(sb.env["GH_TOKEN"], "t") + self.assertNotIn("OTHER", sb.env) + self.assertEqual(sb.env["XDG_CONFIG_HOME"], "/x") + finally: + sb.cleanup() +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `python3 $KIT/selftest.py -v` +Expected: `ModuleNotFoundError: No module named 'lib.isolation'` + +- [ ] **Step 3: Implement** + +`lib/isolation.py`: + +```python +from __future__ import annotations + +import os +import shutil +import subprocess +import tempfile +from dataclasses import dataclass +from pathlib import Path + +ENV_ALLOWLIST = ("PATH", "TERM", "LANG", "LC_ALL", "LC_CTYPE", "TMPDIR", "SHELL", "USER", "LOGNAME") + + +def git(repo: Path, *args: str, env: dict | None = None) -> str: + return subprocess.run( + ["git", "-C", str(repo), *args], check=True, capture_output=True, text=True, env=env, + ).stdout + + +@dataclass +class Sandbox: + root: Path + repo: Path + config_root: Path + env: dict[str, str] + keep: bool = False + + def cleanup(self) -> None: + if not self.keep: + shutil.rmtree(self.root, ignore_errors=True) + + +def new_sandbox( + lever: str, + real_root: Path | None, + credential_files: list[str], + passthrough_env: list[str] = (), + extra_env: dict[str, str] | None = None, + keep: bool = False, + base: Path | None = None, +) -> Sandbox: + root = Path(tempfile.mkdtemp(prefix="skprobe-", dir=base)).resolve() + repo = root / "repo" + repo.mkdir() + git(repo, "init", "-q", "-b", "main") + git(repo, "config", "user.email", "probe@example.invalid") + git(repo, "config", "user.name", "probe") + (repo / "README.md").write_text("probe repo\n") + git(repo, "add", "README.md") + git(repo, "commit", "-q", "-m", "init") + + config_root = root / "config" + config_root.mkdir() + if real_root is not None: + for rel in credential_files: + src = real_root / rel + if src.is_file(): + dst = config_root / rel + dst.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(src, dst) + + env = {k: os.environ[k] for k in ENV_ALLOWLIST if k in os.environ} + for k in passthrough_env: + if k in os.environ: + env[k] = os.environ[k] + env["HOME"] = os.environ.get("HOME", str(root)) + env[lever] = str(config_root) + if extra_env: + env.update(extra_env) + return Sandbox(root=root, repo=repo, config_root=config_root, env=env, keep=keep) +``` + +- [ ] **Step 4: Run to verify pass** + +Run: `python3 $KIT/selftest.py -v` +Expected: 6 tests, `OK` + +- [ ] **Step 5: Commit** + +``` +/usr/bin/git -C add docs/probes/2026-09-16-skills-discovery +/usr/bin/git -C commit -m "Add the probe sandbox with isolated vendor roots (#961)" -m "Co-Authored-By: Claude Fable 5.1 " +``` + +--- + +### Task 4: Git exclusion arms + +**Files:** +- Create: `docs/probes/2026-09-16-skills-discovery/lib/git_exclusion.py` +- Modify: `docs/probes/2026-09-16-skills-discovery/selftest.py` + +**Interfaces:** +- Consumes: `lib.isolation.git`, `lib.isolation.new_sandbox`. +- Produces: + - `EXCLUSIONS = ("none", "gitignore", "info-exclude")`. + - `apply(repo: Path, exclusion: str, rel_dir: str) -> Path | None`: writes the pattern `//` to `.gitignore` (then commits it) or to the file `git rev-parse --git-path info/exclude` names (resolved against `repo` when relative); returns the file written or `None` for `none`. + - `assert_untracked_state(repo: Path, rel_dir: str, exclusion: str) -> str`: runs `git status --porcelain --untracked-files=all -- `; for `none` requires at least one `??` line, otherwise requires empty output; returns the porcelain text; raises `AssertionError` with the porcelain text on mismatch. + +- [ ] **Step 1: Write the failing tests** + +Append to `selftest.py`: + +```python +from lib.git_exclusion import EXCLUSIONS, apply, assert_untracked_state # noqa: E402 + + +class GitExclusionTests(unittest.TestCase): + def _sandbox(self, d): + return new_sandbox("HOME", None, [], base=Path(d)) + + def test_none_shows_untracked(self): + with tempfile.TemporaryDirectory() as d: + sb = self._sandbox(d) + (sb.repo / ".claude/skills/kcap-probe-abc123").mkdir(parents=True) + (sb.repo / ".claude/skills/kcap-probe-abc123/SKILL.md").write_text("x") + self.assertIsNone(apply(sb.repo, "none", ".claude/skills/kcap-probe-abc123")) + out = assert_untracked_state(sb.repo, ".claude/skills/kcap-probe-abc123", "none") + self.assertIn("??", out) + with self.assertRaises(AssertionError): + assert_untracked_state(sb.repo, ".claude/skills/kcap-probe-abc123", "gitignore") + + def test_gitignore_and_info_exclude(self): + for exclusion in ("gitignore", "info-exclude"): + with tempfile.TemporaryDirectory() as d: + sb = self._sandbox(d) + rel = ".agents/skills/kcap-probe-abc123" + (sb.repo / rel).mkdir(parents=True) + (sb.repo / rel / "SKILL.md").write_text("x") + written = apply(sb.repo, exclusion, rel) + self.assertIsNotNone(written) + self.assertIn(f"/{rel}/", written.read_text()) + out = assert_untracked_state(sb.repo, rel, exclusion) + self.assertEqual(out, "") + if exclusion == "gitignore": + self.assertEqual(git(sb.repo, "status", "--porcelain").strip(), "") + else: + self.assertNotIn(".gitignore", git(sb.repo, "ls-files")) + + def test_info_exclude_inside_worktree(self): + with tempfile.TemporaryDirectory() as d: + sb = self._sandbox(d) + wt = sb.root / "wt" + git(sb.repo, "worktree", "add", "-q", "-b", "wt", str(wt)) + rel = ".pi/skills/kcap-probe-abc123" + (wt / rel).mkdir(parents=True) + (wt / rel / "SKILL.md").write_text("x") + written = apply(wt, "info-exclude", rel) + self.assertEqual(written, written.resolve()) + self.assertEqual(assert_untracked_state(wt, rel, "info-exclude"), "") + self.assertIn(EXCLUSIONS[2], "info-exclude") +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `python3 $KIT/selftest.py -v` +Expected: `ModuleNotFoundError: No module named 'lib.git_exclusion'` + +- [ ] **Step 3: Implement** + +`lib/git_exclusion.py`: + +```python +from __future__ import annotations + +from pathlib import Path + +from lib.isolation import git + +EXCLUSIONS = ("none", "gitignore", "info-exclude") + + +def apply(repo: Path, exclusion: str, rel_dir: str) -> Path | None: + pattern = f"/{rel_dir.strip('/')}/\n" + if exclusion == "none": + return None + if exclusion == "gitignore": + target = repo / ".gitignore" + with target.open("a") as f: + f.write(pattern) + git(repo, "add", ".gitignore") + git(repo, "commit", "-q", "-m", "ignore probe skill") + return target + if exclusion == "info-exclude": + raw = git(repo, "rev-parse", "--git-path", "info/exclude").strip() + target = Path(raw) if Path(raw).is_absolute() else repo / raw + target = target.resolve() + target.parent.mkdir(parents=True, exist_ok=True) + with target.open("a") as f: + f.write(pattern) + return target + raise ValueError(exclusion) + + +def assert_untracked_state(repo: Path, rel_dir: str, exclusion: str) -> str: + out = git(repo, "status", "--porcelain", "--untracked-files=all", "--", rel_dir) + if exclusion == "none": + if "??" not in out: + raise AssertionError(f"expected an untracked entry for {rel_dir}, got: {out!r}") + elif out.strip(): + raise AssertionError(f"expected {rel_dir} to be excluded under {exclusion}, got: {out!r}") + return out +``` + +- [ ] **Step 4: Run to verify pass** + +Run: `python3 $KIT/selftest.py -v` +Expected: 9 tests, `OK` + +- [ ] **Step 5: Commit** + +``` +/usr/bin/git -C add docs/probes/2026-09-16-skills-discovery +/usr/bin/git -C commit -m "Add the Git exclusion arms to the probe kit (#961)" -m "Co-Authored-By: Claude Fable 5.1 " +``` + +--- + +### Task 5: Verdict rules + +**Files:** +- Create: `docs/probes/2026-09-16-skills-discovery/lib/verdict.py` +- Modify: `docs/probes/2026-09-16-skills-discovery/selftest.py` + +**Interfaces:** +- Consumes: `lib.probe_skill.Reply`. +- Produces: + - `VERDICTS = ("visible_first_turn", "visible_after_reload", "catalogue_only", "not_visible", "leaked", "untested")`. + - `class PromptDesignFailure(Exception)`. + - `judge_single(expected_token: str, reply: Reply, reload_used: bool = False) -> str`. + - `judge_control(reply: Reply) -> str`: returns `not_visible` when no token; raises `PromptDesignFailure` when any token is present. + - `judge_root(token: str, documented: bool, reply: Reply) -> str`: `visible_first_turn` when found and documented, `leaked` when found and undocumented, else `not_visible`. + - `combine(verdicts: list[str]) -> tuple[str, bool]`: majority verdict and a `flaky` flag (`True` when the runs disagree). Empty input gives `("untested", False)`. + - `needs_third_run(verdicts: list[str]) -> bool`: `True` when exactly two runs disagree. + +- [ ] **Step 1: Write the failing tests** + +Append to `selftest.py`: + +```python +from lib.probe_skill import Reply # noqa: E402 +from lib.verdict import ( # noqa: E402 + VERDICTS, PromptDesignFailure, combine, judge_control, judge_root, judge_single, needs_third_run, +) + + +class VerdictTests(unittest.TestCase): + def test_judge_single(self): + found = Reply(frozenset({"a" * 12}), True, False) + self.assertEqual(judge_single("a" * 12, found), "visible_first_turn") + self.assertEqual(judge_single("a" * 12, found, reload_used=True), "visible_after_reload") + self.assertEqual(judge_single("b" * 12, Reply(frozenset(), True, False)), "catalogue_only") + self.assertEqual(judge_single("b" * 12, Reply(frozenset(), False, True)), "not_visible") + + def test_judge_control(self): + self.assertEqual(judge_control(Reply(frozenset(), False, True)), "not_visible") + with self.assertRaises(PromptDesignFailure): + judge_control(Reply(frozenset({"c" * 12}), False, False)) + + def test_judge_root(self): + r = Reply(frozenset({"d" * 12}), True, False) + self.assertEqual(judge_root("d" * 12, True, r), "visible_first_turn") + self.assertEqual(judge_root("d" * 12, False, r), "leaked") + self.assertEqual(judge_root("e" * 12, True, r), "not_visible") + + def test_combine(self): + self.assertEqual(combine([]), ("untested", False)) + self.assertEqual(combine(["visible_first_turn", "visible_first_turn"]), ("visible_first_turn", False)) + self.assertEqual(combine(["visible_first_turn", "not_visible", "visible_first_turn"]), + ("visible_first_turn", True)) + self.assertTrue(needs_third_run(["visible_first_turn", "not_visible"])) + self.assertFalse(needs_third_run(["not_visible", "not_visible"])) + self.assertFalse(needs_third_run(["not_visible"])) + for v in VERDICTS: + self.assertEqual(combine([v]), (v, False)) +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `python3 $KIT/selftest.py -v` +Expected: `ModuleNotFoundError: No module named 'lib.verdict'` + +- [ ] **Step 3: Implement** + +`lib/verdict.py`: + +```python +from __future__ import annotations + +from collections import Counter + +from lib.probe_skill import Reply + +VERDICTS = ("visible_first_turn", "visible_after_reload", "catalogue_only", "not_visible", "leaked", "untested") + + +class PromptDesignFailure(Exception): + """The negative control produced a token: the prompt or parser is unsound for this entry.""" + + +def judge_single(expected_token: str, reply: Reply, reload_used: bool = False) -> str: + if expected_token in reply.tokens: + return "visible_after_reload" if reload_used else "visible_first_turn" + if reply.skill_named: + return "catalogue_only" + return "not_visible" + + +def judge_control(reply: Reply) -> str: + if reply.tokens: + raise PromptDesignFailure(f"negative control produced tokens {sorted(reply.tokens)}") + return "not_visible" + + +def judge_root(token: str, documented: bool, reply: Reply) -> str: + if token in reply.tokens: + return "visible_first_turn" if documented else "leaked" + return "not_visible" + + +def combine(verdicts: list[str]) -> tuple[str, bool]: + if not verdicts: + return "untested", False + counts = Counter(verdicts) + top, _ = counts.most_common(1)[0] + return top, len(counts) > 1 + + +def needs_third_run(verdicts: list[str]) -> bool: + return len(verdicts) == 2 and verdicts[0] != verdicts[1] +``` + +- [ ] **Step 4: Run to verify pass** + +Run: `python3 $KIT/selftest.py -v` +Expected: 13 tests, `OK` + +- [ ] **Step 5: Commit** + +``` +/usr/bin/git -C add docs/probes/2026-09-16-skills-discovery +/usr/bin/git -C commit -m "Add verdict rules to the probe kit (#961)" -m "Co-Authored-By: Claude Fable 5.1 " +``` + +--- + +### Task 6: Run recorder and matrix emission + +**Files:** +- Create: `docs/probes/2026-09-16-skills-discovery/lib/recorder.py` +- Modify: `docs/probes/2026-09-16-skills-discovery/selftest.py` + +**Interfaces:** +- Consumes: `lib.verdict.combine`. +- Produces: + - `@dataclass class RunRecord` with fields (all JSON-serialisable): `entry: str`, `harness: str`, `binary: str`, `version: str`, `os: str`, `mode: str`, `argv: list[str]`, `isolation_lever: str`, `credential_files: list[str]`, `auth_ok: bool | None`, `scenario: str`, `arm: str`, `root: str | None`, `exclusion: str`, `hook: dict | None` (`{"mechanism", "config_path", "fired_at"}`), `first_request_at: float | None`, `reply: str`, `tokens_found: list[str]`, `skill_named: bool`, `stderr_path: str | None`, `verdict: str`, `duration_ms: int`, `notes: str = ""`, `expected_tokens: dict[str, str]` (root key to token for S4, single key `"native"` otherwise). + - `os_label() -> str` such as `macOS 26.6.2 arm64`. + - `run_dir(outdir: Path, entry: str, mode: str, scenario: str, arm: str) -> Path` (`////`). + - `write_run(outdir: Path, rec: RunRecord) -> Path`: next free `run.json` (N from 1) in `run_dir`. + - `load_runs(outdir: Path) -> list[RunRecord]`. + - `emit_matrix(outdir: Path, target: Path) -> list[dict]`: groups runs by `(entry, mode, scenario, arm, root, exclusion)`, keeps only the highest `version` per entry, applies `combine`, writes rows sorted by that key to `target` as JSON with keys `entry, harness, version, os, mode, scenario, arm, root, exclusion, verdict, flaky, runs, mechanism, evidence, notes` and returns them. `mechanism` is the `hook.mechanism` of the first run, `evidence` the run file paths relative to `target.parent`. + +- [ ] **Step 1: Write the failing tests** + +Append to `selftest.py`: + +```python +import json # noqa: E402 + +from lib.recorder import RunRecord, emit_matrix, load_runs, os_label, run_dir, write_run # noqa: E402 + + +def _rec(**over): + base = dict( + entry="fake", harness="fake", binary="/bin/fake", version="1.0", os=os_label(), mode="print", + argv=["fake", "-p"], isolation_lever="FAKE_HOME", credential_files=[], auth_ok=True, + scenario="S1", arm="S1/native", root=".fake/skills", exclusion="none", + hook=None, first_request_at=1.0, reply="tok", tokens_found=["a" * 12], skill_named=True, + stderr_path=None, verdict="visible_first_turn", duration_ms=10, expected_tokens={"native": "a" * 12}, + ) + base.update(over) + return RunRecord(**base) + + +class RecorderTests(unittest.TestCase): + def test_write_load_emit(self): + with tempfile.TemporaryDirectory() as d: + out = Path(d) / "out" + p1 = write_run(out, _rec()) + p2 = write_run(out, _rec(verdict="not_visible")) + self.assertEqual(p1.name, "run1.json") + self.assertEqual(p2.name, "run2.json") + self.assertEqual(p1.parent, run_dir(out, "fake", "print", "S1", "S1/native")) + write_run(out, _rec(verdict="visible_first_turn")) + write_run(out, _rec(version="0.9", verdict="not_visible", arm="S0/none", scenario="S0")) + self.assertEqual(len(load_runs(out)), 4) + rows = emit_matrix(out, Path(d) / "matrix.json") + self.assertEqual(len(rows), 1) + row = rows[0] + self.assertEqual(row["verdict"], "visible_first_turn") + self.assertTrue(row["flaky"]) + self.assertEqual(row["runs"], 3) + self.assertEqual(row["version"], "1.0") + self.assertTrue(all(e.startswith("out/") for e in row["evidence"])) + self.assertEqual(json.loads((Path(d) / "matrix.json").read_text())[0]["entry"], "fake") + + def test_os_label(self): + self.assertRegex(os_label(), r"^\S+ \S+ \S+$") +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `python3 $KIT/selftest.py -v` +Expected: `ModuleNotFoundError: No module named 'lib.recorder'` + +- [ ] **Step 3: Implement** + +`lib/recorder.py`: + +```python +from __future__ import annotations + +import json +import platform +import subprocess +from dataclasses import asdict, dataclass, field +from pathlib import Path + +from lib.verdict import combine + + +@dataclass +class RunRecord: + entry: str + harness: str + binary: str + version: str + os: str + mode: str + argv: list[str] + isolation_lever: str + credential_files: list[str] + auth_ok: bool | None + scenario: str + arm: str + root: str | None + exclusion: str + hook: dict | None + first_request_at: float | None + reply: str + tokens_found: list[str] + skill_named: bool + stderr_path: str | None + verdict: str + duration_ms: int + expected_tokens: dict[str, str] = field(default_factory=dict) + notes: str = "" + + +def os_label() -> str: + if platform.system() == "Darwin": + ver = subprocess.run(["sw_vers", "-productVersion"], capture_output=True, text=True).stdout.strip() + return f"macOS {ver} {platform.machine()}" + return f"{platform.system()} {platform.release()} {platform.machine()}" + + +def run_dir(outdir: Path, entry: str, mode: str, scenario: str, arm: str) -> Path: + return outdir / entry / mode / scenario / arm.replace("/", "_") + + +def write_run(outdir: Path, rec: RunRecord) -> Path: + d = run_dir(outdir, rec.entry, rec.mode, rec.scenario, rec.arm) + d.mkdir(parents=True, exist_ok=True) + n = 1 + while (d / f"run{n}.json").exists(): + n += 1 + path = d / f"run{n}.json" + path.write_text(json.dumps(asdict(rec), indent=2, sort_keys=True) + "\n") + return path + + +def load_runs(outdir: Path) -> list[RunRecord]: + runs = [] + for path in sorted(outdir.rglob("run*.json")): + data = json.loads(path.read_text()) + data["_path"] = path + rec = RunRecord(**{k: v for k, v in data.items() if k != "_path"}) + rec._path = path # type: ignore[attr-defined] + runs.append(rec) + return runs + + +def _version_key(v: str) -> tuple: + parts = [] + for piece in v.replace("-", ".").split("."): + parts.append((0, int(piece)) if piece.isdigit() else (1, piece)) + return tuple(parts) + + +def emit_matrix(outdir: Path, target: Path) -> list[dict]: + runs = load_runs(outdir) + latest: dict[str, str] = {} + for r in runs: + if r.entry not in latest or _version_key(r.version) > _version_key(latest[r.entry]): + latest[r.entry] = r.version + groups: dict[tuple, list[RunRecord]] = {} + for r in runs: + if r.version != latest[r.entry]: + continue + groups.setdefault((r.entry, r.mode, r.scenario, r.arm, r.root or "", r.exclusion), []).append(r) + rows = [] + for key in sorted(groups): + members = groups[key] + verdict, flaky = combine([m.verdict for m in members]) + first = members[0] + rows.append({ + "entry": first.entry, "harness": first.harness, "version": first.version, "os": first.os, + "mode": first.mode, "scenario": first.scenario, "arm": first.arm, "root": first.root, + "exclusion": first.exclusion, "verdict": verdict, "flaky": flaky, "runs": len(members), + "mechanism": (first.hook or {}).get("mechanism"), + "evidence": [str(m._path.relative_to(target.parent)) for m in members], # type: ignore[attr-defined] + "notes": "; ".join(n for n in {m.notes for m in members} if n), + }) + target.write_text(json.dumps(rows, indent=2) + "\n") + return rows +``` + +- [ ] **Step 4: Run to verify pass** + +Run: `python3 $KIT/selftest.py -v` +Expected: 15 tests, `OK` + +- [ ] **Step 5: Commit** + +``` +/usr/bin/git -C add docs/probes/2026-09-16-skills-discovery +/usr/bin/git -C commit -m "Add the run recorder and matrix emission (#961)" -m "Co-Authored-By: Claude Fable 5.1 " +``` + +--- + +### Task 7: Startup hook payload + +**Files:** +- Create: `docs/probes/2026-09-16-skills-discovery/lib/hook_script.py` +- Modify: `docs/probes/2026-09-16-skills-discovery/selftest.py` + +**Interfaces:** +- Consumes: `lib.probe_skill.ProbeSkill`. +- Produces: + - `write_hook_script(config_root: Path, skill_file: Path, body: str, stamp: Path, delete: bool = False) -> Path`: writes `/probe-hook.sh` (mode 0755) which creates `skill_file`'s parent, writes `body` to `skill_file` (or deletes the parent directory when `delete`), writes `{"fired_at": , "pid": }` to `stamp`, saves up to one second of stdin to `.stdin`, prints nothing, exits 0. + - `read_stamp(stamp: Path) -> dict | None`. + - `stamp_path(config_root: Path) -> Path` = `/probe-hook-fired.json`. + +- [ ] **Step 1: Write the failing tests** + +Append to `selftest.py`: + +```python +import subprocess # noqa: E402 + +from lib.hook_script import read_stamp, stamp_path, write_hook_script # noqa: E402 + + +class HookScriptTests(unittest.TestCase): + def test_script_writes_skill_and_stamp(self): + with tempfile.TemporaryDirectory() as d: + cfg = Path(d) / "cfg" + cfg.mkdir() + skill = ProbeSkill.fresh() + target = Path(d) / "repo" / ".claude" / "skills" / skill.name / "SKILL.md" + stamp = stamp_path(cfg) + script = write_hook_script(cfg, target, skill.render(), stamp) + self.assertEqual(script.stat().st_mode & 0o111, 0o111) + self.assertIsNone(read_stamp(stamp)) + proc = subprocess.run([str(script)], input='{"hook_event_name":"SessionStart"}', + capture_output=True, text=True, timeout=10) + self.assertEqual(proc.returncode, 0, proc.stderr) + self.assertEqual(proc.stdout, "") + self.assertEqual(target.read_text(), skill.render()) + self.assertIn("fired_at", read_stamp(stamp)) + self.assertEqual(Path(str(stamp) + ".stdin").read_text(), '{"hook_event_name":"SessionStart"}') + + def test_script_survives_open_stdin_and_can_delete(self): + with tempfile.TemporaryDirectory() as d: + cfg = Path(d) / "cfg" + cfg.mkdir() + skill = ProbeSkill.fresh() + target = Path(d) / "repo" / ".x" / skill.name / "SKILL.md" + target.parent.mkdir(parents=True) + target.write_text("old") + stamp = stamp_path(cfg) + script = write_hook_script(cfg, target, "", stamp, delete=True) + with subprocess.Popen([str(script)], stdin=subprocess.PIPE, stdout=subprocess.PIPE) as p: + out, _ = p.communicate(timeout=10) + self.assertEqual(p.returncode, 0) + self.assertFalse(target.parent.exists()) + self.assertIsNotNone(read_stamp(stamp)) +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `python3 $KIT/selftest.py -v` +Expected: `ModuleNotFoundError: No module named 'lib.hook_script'` + +- [ ] **Step 3: Implement** + +`lib/hook_script.py`: + +```python +from __future__ import annotations + +import json +from pathlib import Path + + +def stamp_path(config_root: Path) -> Path: + return config_root / "probe-hook-fired.json" + + +def write_hook_script(config_root: Path, skill_file: Path, body: str, stamp: Path, delete: bool = False) -> Path: + if delete: + action = f"rm -rf '{skill_file.parent}'\n" + else: + action = ( + f"mkdir -p '{skill_file.parent}'\n" + f"cat > '{skill_file}' <<'KCAP_PROBE_EOF'\n{body}KCAP_PROBE_EOF\n" + ) + script = config_root / "probe-hook.sh" + script.write_text( + "#!/bin/sh\n" + "set -eu\n" + f"{action}" + f"printf '{{\"fired_at\": %s, \"pid\": %s}}\\n' \"$(date +%s)\" \"$$\" > '{stamp}'\n" + # A vendor that never closes the hook's stdin must not wedge the launch: capture briefly. + f"( cat > '{stamp}.stdin' ) & cat_pid=$!\n" + "sleep 1\n" + "kill $cat_pid 2>/dev/null || true\n" + "exit 0\n" + ) + script.chmod(0o755) + return script + + +def read_stamp(stamp: Path) -> dict | None: + if not stamp.exists(): + return None + return json.loads(stamp.read_text()) +``` + +- [ ] **Step 4: Run to verify pass** + +Run: `python3 $KIT/selftest.py -v` +Expected: 17 tests, `OK`. If `test_script_writes_skill_and_stamp` reports an empty `.stdin` file, the background `cat` lost the race to `kill`: raise `sleep 1` to `sleep 2` in the script and re-run. + +- [ ] **Step 5: Commit** + +``` +/usr/bin/git -C add docs/probes/2026-09-16-skills-discovery +/usr/bin/git -C commit -m "Add the startup hook payload to the probe kit (#961)" -m "Co-Authored-By: Claude Fable 5.1 " +``` + +--- + +### Task 8: Adapter contract, fake adapter, print driver + +**Files:** +- Create: `docs/probes/2026-09-16-skills-discovery/harness/base.py` +- Create: `docs/probes/2026-09-16-skills-discovery/harness/fake.py` +- Create: `docs/probes/2026-09-16-skills-discovery/lib/print_driver.py` +- Modify: `docs/probes/2026-09-16-skills-discovery/selftest.py` + +**Interfaces:** +- Consumes: `lib.isolation.Sandbox`, `lib.hook_script`, `lib.probe_skill`. +- Produces (`harness/base.py`): + - `@dataclass class HookInfo(mechanism: str, config_path: str)`. + - `@dataclass class AskResult(reply_text: str, raw: str, argv: list[str], started_at: float, first_request_at: float, stderr_path: str | None, exit_code: int | None)`. + - `class Adapter` with class attributes `entry: str`, `harness: str`, `binary: str`, `lever: str`, `credential_files: list[str] = []`, `passthrough_env: list[str] = []`, `extra_env: dict[str, str] = {}`, `native_root: str`, `documented_roots: frozenset[str]`, `flat_skill_layout: bool = False`, `modes: tuple[str, ...] = ("print", "daemon")`, `turn_timeout: float = 180.0`; methods `binary_path() -> str | None` (`shutil.which`), `version() -> str` (runs ` --version`, first line, stripped), `real_root() -> Path | None`, `prepare(sb: Sandbox) -> None` (default no-op), `check_auth(sb: Sandbox) -> bool | None` (default `None`), `install_startup_hook(sb: Sandbox, script: Path) -> HookInfo` (abstract), `install_registration(sb: Sandbox, skill_file: Path, body: str) -> HookInfo | None` (default `None`), `ask(sb: Sandbox, mode: str, prompt: str) -> AskResult` (abstract), `list_catalogue(sb: Sandbox) -> str | None` (default `None`), `skill_dir(sb: Sandbox, root: str, name: str) -> Path`, `skill_file(sb: Sandbox, root: str, name: str) -> Path` (honours `flat_skill_layout`). +- Produces (`lib/print_driver.py`): `print_ask(argv: list[str], cwd: Path, env: dict, stderr_path: Path, timeout: float, stdin_text: str | None = None, extract: Callable[[str], str] | None = None) -> AskResult`: runs the process, captures stdout to `raw`, `reply_text = extract(raw)` when given else `raw`, records `started_at` and `first_request_at` (both the spawn instant for print mode), kills on timeout and records `exit_code=None`. +- Produces (`harness/fake.py`): `class FakeAdapter(Adapter)` whose `ask` runs the installed hook script if one was installed, then reads every `kcap-probe-*/SKILL.md` under the roots in `documented_roots` inside `sb.repo` and returns `"="` lines, or `NO-SKILL`. Entry `fake`, native root `.fake/skills`, documented roots `{".fake/skills", ".agents/skills"}`, lever `FAKE_HOME`, binary `sh`. + +- [ ] **Step 1: Write the failing tests** + +Append to `selftest.py`: + +```python +from harness.base import Adapter, AskResult, HookInfo # noqa: E402 +from harness.fake import FakeAdapter # noqa: E402 +from lib.print_driver import print_ask # noqa: E402 + + +class AdapterTests(unittest.TestCase): + def test_fake_adapter_round_trip(self): + with tempfile.TemporaryDirectory() as d: + fa = FakeAdapter() + sb = new_sandbox(fa.lever, None, [], base=Path(d)) + skill = ProbeSkill.fresh() + write_skill(sb.repo / fa.native_root, skill) + res = fa.ask(sb, "print", single_prompt(skill)) + self.assertIsInstance(res, AskResult) + self.assertIn(skill.body_token, res.reply_text) + self.assertIn(skill.body_token, res.raw) + self.assertEqual(fa.skill_file(sb, fa.native_root, skill.name), + sb.repo / ".fake/skills" / skill.name / "SKILL.md") + + def test_fake_adapter_runs_hook_before_reading(self): + with tempfile.TemporaryDirectory() as d: + fa = FakeAdapter() + sb = new_sandbox(fa.lever, None, [], base=Path(d)) + skill = ProbeSkill.fresh() + script = write_hook_script(sb.config_root, fa.skill_file(sb, fa.native_root, skill.name), + skill.render(), stamp_path(sb.config_root)) + info = fa.install_startup_hook(sb, script) + self.assertIsInstance(info, HookInfo) + res = fa.ask(sb, "print", single_prompt(skill)) + self.assertIn(skill.body_token, res.reply_text) + self.assertIsNotNone(read_stamp(stamp_path(sb.config_root))) + + def test_print_driver_timeout_and_extract(self): + with tempfile.TemporaryDirectory() as d: + res = print_ask(["sh", "-c", "echo '{\"result\":\"hi\"}'"], Path(d), dict(os.environ), + Path(d) / "e.stderr.log", timeout=10, + extract=lambda raw: json.loads(raw)["result"]) + self.assertEqual(res.reply_text, "hi") + self.assertEqual(res.exit_code, 0) + slow = print_ask(["sh", "-c", "sleep 5"], Path(d), dict(os.environ), Path(d) / "s.stderr.log", timeout=0.5) + self.assertIsNone(slow.exit_code) + + def test_adapter_defaults(self): + self.assertIsNone(Adapter.check_auth(FakeAdapter(), None)) + self.assertIsNone(Adapter.install_registration(FakeAdapter(), None, None, None)) +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `python3 $KIT/selftest.py -v` +Expected: `ModuleNotFoundError: No module named 'harness.base'` + +- [ ] **Step 3: Implement the contract** + +`harness/base.py`: + +```python +from __future__ import annotations + +import shutil +import subprocess +from dataclasses import dataclass, field +from pathlib import Path + +from lib.isolation import Sandbox + + +@dataclass +class HookInfo: + mechanism: str + config_path: str + + +@dataclass +class AskResult: + reply_text: str + raw: str + argv: list[str] + started_at: float + first_request_at: float + stderr_path: str | None + exit_code: int | None + notes: str = "" + + +class Adapter: + entry: str = "" + harness: str = "" + binary: str = "" + lever: str = "" + credential_files: list[str] = [] + passthrough_env: list[str] = [] + extra_env: dict[str, str] = {} + native_root: str = "" + documented_roots: frozenset[str] = frozenset() + flat_skill_layout: bool = False + modes: tuple[str, ...] = ("print", "daemon") + turn_timeout: float = 180.0 + + def binary_path(self) -> str | None: + return shutil.which(self.binary) + + def version(self) -> str: + path = self.binary_path() + if path is None: + return "not-installed" + out = subprocess.run([path, "--version"], capture_output=True, text=True, timeout=60) + return (out.stdout or out.stderr).strip().splitlines()[0] if (out.stdout or out.stderr).strip() else "unknown" + + def real_root(self) -> Path | None: + return None + + def prepare(self, sb: Sandbox) -> None: + return None + + def check_auth(self, sb: Sandbox) -> bool | None: + return None + + def install_startup_hook(self, sb: Sandbox, script: Path) -> HookInfo: + raise NotImplementedError + + def install_registration(self, sb: Sandbox, skill_file: Path, body: str) -> HookInfo | None: + return None + + def ask(self, sb: Sandbox, mode: str, prompt: str) -> AskResult: + raise NotImplementedError + + def list_catalogue(self, sb: Sandbox) -> str | None: + return None + + def skill_dir(self, sb: Sandbox, root: str, name: str) -> Path: + return sb.repo / root if self.flat_skill_layout else sb.repo / root / name + + def skill_file(self, sb: Sandbox, root: str, name: str) -> Path: + if self.flat_skill_layout: + return sb.repo / root / f"{name}.md" + return sb.repo / root / name / "SKILL.md" +``` + +`lib/print_driver.py`: + +```python +from __future__ import annotations + +import subprocess +import time +from pathlib import Path +from typing import Callable + +from harness.base import AskResult + + +def print_ask( + argv: list[str], cwd: Path, env: dict, stderr_path: Path, timeout: float, + stdin_text: str | None = None, extract: Callable[[str], str] | None = None, +) -> AskResult: + started = time.time() + with stderr_path.open("ab") as err: + try: + proc = subprocess.run( + argv, cwd=str(cwd), env=env, input=stdin_text, capture_output=False, + stdout=subprocess.PIPE, stderr=err, text=True, timeout=timeout, + stdin=None if stdin_text is not None else subprocess.DEVNULL, + ) + raw, code = proc.stdout, proc.returncode + except subprocess.TimeoutExpired as ex: + raw = (ex.stdout or b"").decode("utf-8", "replace") if isinstance(ex.stdout, bytes) else (ex.stdout or "") + code = None + reply = raw + if extract is not None: + try: + reply = extract(raw) + except Exception: # noqa: BLE001 + reply = raw + return AskResult(reply_text=reply, raw=raw, argv=list(argv), started_at=started, + first_request_at=started, stderr_path=str(stderr_path), exit_code=code) +``` + +`harness/fake.py`: + +```python +from __future__ import annotations + +import subprocess +import time +from pathlib import Path + +from harness.base import Adapter, AskResult, HookInfo +from lib.isolation import Sandbox +from lib.probe_skill import NO_SKILL, TOKEN_RE + + +class FakeAdapter(Adapter): + entry = "fake" + harness = "fake" + binary = "sh" + lever = "FAKE_HOME" + native_root = ".fake/skills" + documented_roots = frozenset({".fake/skills", ".agents/skills"}) + + def __init__(self) -> None: + self._hook: Path | None = None + + def version(self) -> str: + return "1.0" + + def install_startup_hook(self, sb: Sandbox, script: Path) -> HookInfo: + self._hook = script + return HookInfo(mechanism="fake-startup", config_path=str(script)) + + def ask(self, sb: Sandbox, mode: str, prompt: str) -> AskResult: + started = time.time() + if self._hook is not None: + subprocess.run([str(self._hook)], input="{}", capture_output=True, text=True, timeout=10) + lines = [] + for root in sorted(self.documented_roots): + for skill_md in sorted((sb.repo / root).glob("kcap-probe-*/SKILL.md")): + m = TOKEN_RE.search(skill_md.read_text()) + if m: + lines.append(f"{skill_md.parent.name}=PROBE-BODY-{m.group(1)}") + text = "\n".join(lines) if lines else NO_SKILL + return AskResult(reply_text=text, raw=text, argv=["fake"], started_at=started, + first_request_at=started, stderr_path=None, exit_code=0) +``` + +- [ ] **Step 4: Run to verify pass** + +Run: `python3 $KIT/selftest.py -v` +Expected: 21 tests, `OK` + +- [ ] **Step 5: Commit** + +``` +/usr/bin/git -C add docs/probes/2026-09-16-skills-discovery +/usr/bin/git -C commit -m "Add the adapter contract, fake adapter and print driver (#961)" -m "Co-Authored-By: Claude Fable 5.1 " +``` + +--- + +### Task 9: Orchestrator with scenarios S0 to S4 + +**Files:** +- Create: `docs/probes/2026-09-16-skills-discovery/probe.py` +- Modify: `docs/probes/2026-09-16-skills-discovery/harness/__init__.py` +- Modify: `docs/probes/2026-09-16-skills-discovery/selftest.py` + +**Interfaces:** +- Consumes: everything above. +- Produces (`probe.py`, importable as a module and runnable as a script): + - `ALL_ROOTS: dict[str, str]` mapping root key to root: `{"claude": ".claude/skills", "agents": ".agents/skills", "codex": ".codex/skills", "cursor": ".cursor/skills", "github": ".github/skills", "gemini": ".gemini/skills", "kiro": ".kiro/skills", "pi": ".pi/skills", "opencode": ".opencode/skills", "agent": ".agent/skills"}`. + - `class Runner(adapter: Adapter, outdir: Path, runs: int = 2, keep: bool = False, base: Path | None = None)` with `run_scenario(mode: str, scenario: str, arms: list[str] | None = None) -> list[RunRecord]` and the per-arm methods `arm_s0`, `arm_s1(exclusion="none")`, `arm_s2(arm)`, `arm_s3(exclusion)`, `arm_s4_all()`, `arm_s4_confirm(root_key)`, each returning one `RunRecord` per run and writing it via `write_run`. Repetition rule inside `run_arm(fn) -> list[RunRecord]`: run `self.runs` times; if exactly two disagree, run a third. + - S1 gate: `Runner.s1_ok: dict[str, bool]` per mode; when S1's combined verdict is not `visible_first_turn`, later scenarios in that mode write a single `untested` record with `notes="S1 failed"` instead of spending turns. + - `main(argv: list[str] | None = None) -> int` with the CLI from the spec: `--harness` (repeatable), `--mode`, `--scenario` (repeatable), `--turn`, `--runs`, `--outdir`, `--keep`, `--emit`. Without `--turn` only the free phase runs: version, `binary_path`, `check_auth` in a sandbox, and a `free.json` written under `run_dir(outdir, entry, mode, "free", "free")`. + - `harness/__init__.py`: `ENTRIES: dict[str, type[Adapter]]`, filled by later tasks; Task 9 registers only `fake` under the key `fake`. + +- [ ] **Step 1: Write the failing tests** + +Append to `selftest.py`: + +```python +import probe # noqa: E402 +from lib.recorder import load_runs as _load # noqa: E402 + + +class RunnerTests(unittest.TestCase): + def _runner(self, d, runs=2): + return probe.Runner(FakeAdapter(), Path(d) / "out", runs=runs, base=Path(d)) + + def test_s0_and_s1(self): + with tempfile.TemporaryDirectory() as d: + r = self._runner(d) + recs = r.run_scenario("print", "S0") + self.assertEqual([x.verdict for x in recs], ["not_visible", "not_visible"]) + recs = r.run_scenario("print", "S1") + self.assertEqual([x.verdict for x in recs], ["visible_first_turn"] * 2) + self.assertTrue(r.s1_ok["print"]) + self.assertEqual(recs[0].root, ".fake/skills") + self.assertEqual(recs[0].expected_tokens.keys(), {"native"}) + self.assertEqual(len(_load(Path(d) / "out")), 4) + + def test_s2_hook_arms_and_registration_skip(self): + with tempfile.TemporaryDirectory() as d: + r = self._runner(d) + r.run_scenario("print", "S1") + recs = r.run_scenario("print", "S2") + by_arm = {} + for x in recs: + by_arm.setdefault(x.arm, []).append(x) + self.assertEqual({x.verdict for x in by_arm["S2/hook-creates-root"]}, {"visible_first_turn"}) + self.assertEqual({x.verdict for x in by_arm["S2/hook-adds-skill"]}, {"visible_first_turn"}) + self.assertEqual({x.verdict for x in by_arm["S2/registration"]}, {"untested"}) + self.assertIsNotNone(by_arm["S2/hook-creates-root"][0].hook["fired_at"]) + self.assertEqual(by_arm["S2/hook-creates-root"][0].hook["mechanism"], "fake-startup") + + def test_s3_exclusions(self): + with tempfile.TemporaryDirectory() as d: + r = self._runner(d) + r.run_scenario("print", "S1") + recs = r.run_scenario("print", "S3") + self.assertEqual({x.exclusion for x in recs}, {"gitignore", "info-exclude"}) + self.assertEqual({x.verdict for x in recs}, {"visible_first_turn"}) + + def test_s4_roots_and_confirmation(self): + with tempfile.TemporaryDirectory() as d: + r = self._runner(d) + r.run_scenario("print", "S1") + recs = r.run_scenario("print", "S4") + all_rows = [x for x in recs if x.arm == "S4/all-roots"] + self.assertEqual(len(all_rows), 2) + self.assertEqual(all_rows[0].verdict, "visible_first_turn") + per_root = {} + for x in recs: + if x.arm.startswith("S4/confirm-"): + per_root[x.root] = x.verdict + self.assertEqual(per_root[".claude/skills"], "not_visible") + self.assertNotIn(".fake/skills", per_root) + self.assertNotIn(".agents/skills", per_root) + self.assertEqual(all_rows[0].expected_tokens.keys(), set(probe.ALL_ROOTS) | {"fake"}) + + def test_s1_gate_blocks_later_scenarios(self): + with tempfile.TemporaryDirectory() as d: + r = self._runner(d) + r.s1_ok["print"] = False + recs = r.run_scenario("print", "S3") + self.assertEqual([x.verdict for x in recs], ["untested", "untested"]) + self.assertEqual(recs[0].notes, "S1 failed") + + def test_third_run_on_disagreement(self): + with tempfile.TemporaryDirectory() as d: + r = self._runner(d) + calls = {"n": 0} + + def flaky(): + calls["n"] += 1 + v = "visible_first_turn" if calls["n"] != 2 else "not_visible" + return probe.RunRecord(entry="fake", harness="fake", binary="x", version="1", os="o", + mode="print", argv=[], isolation_lever="L", credential_files=[], + auth_ok=None, scenario="S1", arm="S1/native", root=None, + exclusion="none", hook=None, first_request_at=None, reply="", + tokens_found=[], skill_named=False, stderr_path=None, verdict=v, + duration_ms=0) + recs = r.run_arm(flaky) + self.assertEqual(len(recs), 3) + + def test_cli_free_phase_and_emit(self): + with tempfile.TemporaryDirectory() as d: + out = Path(d) / "out" + code = probe.main(["--harness", "fake", "--mode", "print", "--outdir", str(out), "--base", d]) + self.assertEqual(code, 0) + self.assertTrue((out / "fake" / "print" / "free" / "free" / "free.json").exists()) + code = probe.main(["--harness", "fake", "--mode", "print", "--scenario", "S1", "--turn", + "--outdir", str(out), "--base", d, "--matrix", str(Path(d) / "m.json")]) + self.assertEqual(code, 0) + code = probe.main(["--emit", "--outdir", str(out), "--matrix", str(Path(d) / "m.json")]) + self.assertEqual(code, 0) + rows = json.loads((Path(d) / "m.json").read_text()) + self.assertEqual({r["scenario"] for r in rows}, {"S1"}) +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `python3 $KIT/selftest.py -v` +Expected: `ModuleNotFoundError: No module named 'probe'` + +- [ ] **Step 3: Implement** + +`harness/__init__.py`: + +```python +from __future__ import annotations + +from harness.base import Adapter +from harness.fake import FakeAdapter + +ENTRIES: dict[str, type[Adapter]] = { + "fake": FakeAdapter, +} +``` + +`probe.py`: + +```python +#!/usr/bin/env python3 +"""Skills discovery probes: does a repo-local skill written at startup reach the first model request? + +Usage: + probe.py [--harness ENTRY ...] [--mode print|daemon] [--scenario S0..S4 ...] [--turn] + [--runs N] [--outdir DIR] [--keep] [--emit] [--matrix FILE] [--base DIR] + +Without --turn only the free phase runs (binary, version, auth in an isolated root): zero model +requests. Each turn arm costs one model request per run. +""" +from __future__ import annotations + +import argparse +import json +import sys +import time +from pathlib import Path +from typing import Callable + +KIT = Path(__file__).resolve().parent +sys.path.insert(0, str(KIT)) + +from harness import ENTRIES # noqa: E402 +from harness.base import Adapter, AskResult # noqa: E402 +from lib.git_exclusion import apply as apply_exclusion, assert_untracked_state # noqa: E402 +from lib.hook_script import read_stamp, stamp_path, write_hook_script # noqa: E402 +from lib.isolation import Sandbox, new_sandbox # noqa: E402 +from lib.probe_skill import ProbeSkill, multi_prompt, parse_reply, single_prompt, write_skill # noqa: E402 +from lib.recorder import RunRecord, emit_matrix, os_label, run_dir, write_run # noqa: E402 +from lib.verdict import judge_control, judge_root, judge_single, needs_third_run, combine # noqa: E402 + +ALL_ROOTS: dict[str, str] = { + "claude": ".claude/skills", "agents": ".agents/skills", "codex": ".codex/skills", + "cursor": ".cursor/skills", "github": ".github/skills", "gemini": ".gemini/skills", + "kiro": ".kiro/skills", "pi": ".pi/skills", "opencode": ".opencode/skills", "agent": ".agent/skills", +} +SCENARIOS = ("S0", "S1", "S2", "S3", "S4") +S2_ARMS = ("hook-creates-root", "hook-adds-skill", "registration") +S3_ARMS = ("gitignore", "info-exclude") + + +class Runner: + def __init__(self, adapter: Adapter, outdir: Path, runs: int = 2, keep: bool = False, + base: Path | None = None) -> None: + self.adapter = adapter + self.outdir = outdir + self.runs = runs + self.keep = keep + self.base = base + self.s1_ok: dict[str, bool] = {} + self._version = adapter.version() + self._binary = adapter.binary_path() or adapter.binary + self._os = os_label() + + # -- sandbox plumbing --------------------------------------------------------------------- + + def sandbox(self) -> Sandbox: + a = self.adapter + sb = new_sandbox(a.lever, a.real_root(), a.credential_files, a.passthrough_env, + dict(a.extra_env), keep=self.keep, base=self.base) + a.prepare(sb) + return sb + + def record(self, mode: str, scenario: str, arm: str, root: str | None, exclusion: str, + res: AskResult | None, verdict: str, expected: dict[str, str], hook: dict | None = None, + sb: Sandbox | None = None, notes: str = "", started: float | None = None, + auth_ok: bool | None = None) -> RunRecord: + reply = parse_reply(res.reply_text, res.raw) if res else None + rec = RunRecord( + entry=self.adapter.entry, harness=self.adapter.harness, binary=self._binary, + version=self._version, os=self._os, mode=mode, argv=res.argv if res else [], + isolation_lever=self.adapter.lever, credential_files=list(self.adapter.credential_files), + auth_ok=auth_ok, scenario=scenario, arm=arm, root=root, exclusion=exclusion, hook=hook, + first_request_at=res.first_request_at if res else None, reply=res.reply_text if res else "", + tokens_found=sorted(reply.tokens) if reply else [], skill_named=reply.skill_named if reply else False, + stderr_path=res.stderr_path if res else None, verdict=verdict, + duration_ms=int((time.time() - (started or time.time())) * 1000), + expected_tokens=expected, notes=(notes + ("" if not res or not res.notes else f" {res.notes}")).strip(), + ) + write_run(self.outdir, rec) + return rec + + def run_arm(self, fn: Callable[[], RunRecord]) -> list[RunRecord]: + recs = [fn() for _ in range(self.runs)] + if needs_third_run([r.verdict for r in recs]): + recs.append(fn()) + return recs + + def _blocked(self, mode: str, scenario: str, arm: str, root: str | None, exclusion: str) -> RunRecord: + return self.record(mode, scenario, arm, root, exclusion, None, "untested", {}, notes="S1 failed") + + def _ask(self, sb: Sandbox, mode: str, prompt: str) -> AskResult: + return self.adapter.ask(sb, mode, prompt) + + def _hook_dict(self, sb: Sandbox, mechanism: str, config_path: str) -> dict: + stamp = read_stamp(stamp_path(sb.config_root)) + return {"mechanism": mechanism, "config_path": config_path, + "fired_at": stamp.get("fired_at") if stamp else None} + + # -- arms --------------------------------------------------------------------------------- + + def arm_s0(self, mode: str) -> RunRecord: + started = time.time() + sb = self.sandbox() + try: + skill = ProbeSkill.fresh() + res = self._ask(sb, mode, single_prompt(skill)) + verdict = judge_control(parse_reply(res.reply_text, res.raw)) + return self.record(mode, "S0", "S0/none", None, "none", res, verdict, {}, started=started) + finally: + sb.cleanup() + + def arm_s1(self, mode: str, exclusion: str = "none", scenario: str = "S1", + root: str | None = None) -> RunRecord: + started = time.time() + a = self.adapter + root = root or a.native_root + sb = self.sandbox() + try: + skill = ProbeSkill.fresh() + write_skill(sb.repo / root, skill, flat=a.flat_skill_layout) + rel = str(a.skill_dir(sb, root, skill.name).relative_to(sb.repo)) + apply_exclusion(sb.repo, exclusion, rel) + notes = "" + try: + assert_untracked_state(sb.repo, rel, exclusion) + except AssertionError as ex: + notes = f"git state: {ex}" + res = self._ask(sb, mode, single_prompt(skill)) + verdict = judge_single(skill.token, parse_reply(res.reply_text, res.raw)) + if notes: + verdict = "untested" + arm = "S1/native" if scenario == "S1" else f"S3/{exclusion}" + return self.record(mode, scenario, arm, root, exclusion, res, verdict, + {"native": skill.token}, started=started, notes=notes) + finally: + sb.cleanup() + + def arm_s2(self, mode: str, arm: str) -> RunRecord: + started = time.time() + a = self.adapter + root = a.native_root + sb = self.sandbox() + try: + skill = ProbeSkill.fresh() + target = a.skill_file(sb, root, skill.name) + if arm == "hook-adds-skill": + write_skill(sb.repo / root, ProbeSkill.fresh(), flat=a.flat_skill_layout) + if arm == "registration": + info = a.install_registration(sb, target, skill.render()) + if info is None: + return self.record(mode, "S2", "S2/registration", root, "none", None, "untested", {}, + notes="no registration mechanism for this entry", started=started) + reload_used = True + else: + script = write_hook_script(sb.config_root, target, skill.render(), stamp_path(sb.config_root)) + info = a.install_startup_hook(sb, script) + reload_used = False + res = self._ask(sb, mode, single_prompt(skill)) + verdict = judge_single(skill.token, parse_reply(res.reply_text, res.raw), reload_used=reload_used) + hook = self._hook_dict(sb, info.mechanism, info.config_path) + notes = "" if hook["fired_at"] is not None or arm == "registration" else "hook never fired" + if not target.exists(): + notes = (notes + " skill file absent after the turn").strip() + return self.record(mode, "S2", f"S2/{arm}", root, "none", res, verdict, {"native": skill.token}, + hook=hook, started=started, notes=notes) + finally: + sb.cleanup() + + def arm_s3(self, mode: str, exclusion: str) -> RunRecord: + return self.arm_s1(mode, exclusion=exclusion, scenario="S3") + + def arm_s4_all(self, mode: str) -> RunRecord: + started = time.time() + a = self.adapter + sb = self.sandbox() + try: + roots = dict(ALL_ROOTS) + if a.native_root not in roots.values(): + roots[a.entry.replace("-", "_")] = a.native_root + skills = {key: ProbeSkill.fresh() for key in roots} + for key, skill in skills.items(): + write_skill(sb.repo / roots[key], skill, flat=a.flat_skill_layout and roots[key] == a.native_root) + res = self._ask(sb, mode, multi_prompt()) + reply = parse_reply(res.reply_text, res.raw) + found = {key for key, s in skills.items() if s.token in reply.tokens} + leaked = {key for key in found if roots[key] not in a.documented_roots} + verdict = "leaked" if leaked else ("visible_first_turn" if found else "not_visible") + notes = f"found={sorted(found)} leaked={sorted(leaked)}" + return self.record(mode, "S4", "S4/all-roots", None, "none", res, verdict, + {k: s.token for k, s in skills.items()}, started=started, notes=notes) + finally: + sb.cleanup() + + def arm_s4_confirm(self, mode: str, root_key: str, root: str) -> RunRecord: + started = time.time() + a = self.adapter + sb = self.sandbox() + try: + skill = ProbeSkill.fresh() + write_skill(sb.repo / root, skill, flat=a.flat_skill_layout and root == a.native_root) + res = self._ask(sb, mode, single_prompt(skill)) + verdict = judge_root(skill.token, root in a.documented_roots, parse_reply(res.reply_text, res.raw)) + return self.record(mode, "S4", f"S4/confirm-{root_key}", root, "none", res, verdict, + {root_key: skill.token}, started=started) + finally: + sb.cleanup() + + # -- scenarios ---------------------------------------------------------------------------- + + def run_scenario(self, mode: str, scenario: str, arms: list[str] | None = None) -> list[RunRecord]: + gated = scenario not in ("S0", "S1") and not self.s1_ok.get(mode, True) + out: list[RunRecord] = [] + if scenario == "S0": + out += self.run_arm(lambda: self.arm_s0(mode)) + elif scenario == "S1": + recs = self.run_arm(lambda: self.arm_s1(mode)) + self.s1_ok[mode] = combine([r.verdict for r in recs])[0] == "visible_first_turn" + out += recs + elif scenario == "S2": + for arm in arms or S2_ARMS: + if gated: + out += self.run_arm(lambda: self._blocked(mode, "S2", f"S2/{arm}", self.adapter.native_root, "none")) + elif arm == "registration": + first = self.arm_s2(mode, arm) + out.append(first) + if first.verdict != "untested": + recs = [first, self.arm_s2(mode, arm)] + if needs_third_run([r.verdict for r in recs]): + recs.append(self.arm_s2(mode, arm)) + out += recs[1:] + else: + out += self.run_arm(lambda a=arm: self.arm_s2(mode, a)) + elif scenario == "S3": + for exclusion in arms or S3_ARMS: + if gated: + out += self.run_arm(lambda e=exclusion: self._blocked(mode, "S3", f"S3/{e}", self.adapter.native_root, e)) + else: + out += self.run_arm(lambda e=exclusion: self.arm_s3(mode, e)) + elif scenario == "S4": + if gated: + out += self.run_arm(lambda: self._blocked(mode, "S4", "S4/all-roots", None, "none")) + return out + recs = self.run_arm(lambda: self.arm_s4_all(mode)) + out += recs + roots = dict(ALL_ROOTS) + if self.adapter.native_root not in roots.values(): + roots[self.adapter.entry.replace("-", "_")] = self.adapter.native_root + for key, root in roots.items(): + seen_every_run = all(recs[i].expected_tokens.get(key) in recs[i].tokens_found for i in range(len(recs))) + if seen_every_run: + continue + out += self.run_arm(lambda k=key, r=root: self.arm_s4_confirm(mode, k, r)) + else: + raise ValueError(scenario) + return out + + +def free_phase(adapter: Adapter, outdir: Path, mode: str, base: Path | None) -> dict: + sb = new_sandbox(adapter.lever, adapter.real_root(), adapter.credential_files, adapter.passthrough_env, + dict(adapter.extra_env), base=base) + try: + adapter.prepare(sb) + info = { + "entry": adapter.entry, "binary": adapter.binary_path(), "version": adapter.version(), + "os": os_label(), "mode": mode, "isolation_lever": adapter.lever, + "credential_files": [c for c in adapter.credential_files if (sb.config_root / c).exists()], + "auth_ok": adapter.check_auth(sb) if adapter.binary_path() else None, + "catalogue": adapter.list_catalogue(sb) if adapter.binary_path() else None, + } + finally: + sb.cleanup() + d = run_dir(outdir, adapter.entry, mode, "free", "free") + d.mkdir(parents=True, exist_ok=True) + (d / "free.json").write_text(json.dumps(info, indent=2) + "\n") + return info + + +def main(argv: list[str] | None = None) -> int: + p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + p.add_argument("--harness", action="append", choices=sorted(ENTRIES)) + p.add_argument("--mode", default="print", choices=("print", "daemon")) + p.add_argument("--scenario", action="append", choices=SCENARIOS) + p.add_argument("--turn", action="store_true") + p.add_argument("--runs", type=int, default=2) + p.add_argument("--outdir", type=Path, default=KIT / "out") + p.add_argument("--matrix", type=Path, default=KIT / "matrix.json") + p.add_argument("--keep", action="store_true") + p.add_argument("--emit", action="store_true") + p.add_argument("--base", type=Path, default=None) + args = p.parse_args(argv) + + if args.emit: + rows = emit_matrix(args.outdir, args.matrix) + print(f"wrote {len(rows)} rows to {args.matrix}") + return 0 + + entries = args.harness or [e for e in ENTRIES if e != "fake"] + for name in entries: + adapter = ENTRIES[name]() + if args.mode not in adapter.modes: + print(f"{name}: mode {args.mode} unsupported, skipping") + continue + info = free_phase(adapter, args.outdir, args.mode, args.base) + print(f"{name}: {info['version']} auth_ok={info['auth_ok']}") + if not args.turn: + continue + if info["binary"] is None: + print(f"{name}: binary not installed, turn arms skipped") + continue + runner = Runner(adapter, args.outdir, runs=args.runs, keep=args.keep, base=args.base) + for scenario in args.scenario or SCENARIOS: + recs = runner.run_scenario(args.mode, scenario) + for r in recs: + print(f"{name} {args.mode} {r.arm} root={r.root} excl={r.exclusion} -> {r.verdict} {r.notes}") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) +``` + +- [ ] **Step 4: Run to verify pass** + +Run: `python3 $KIT/selftest.py -v` +Expected: 28 tests, `OK` + +- [ ] **Step 5: Smoke the CLI by hand** + +Run: `python3 $KIT/probe.py --harness fake --mode print --scenario S0 --scenario S1 --turn --outdir /tmp/skprobe-smoke --base /tmp` +Expected: lines ending in `-> not_visible` for S0 and `-> visible_first_turn` for S1. + +- [ ] **Step 6: Commit** + +``` +/usr/bin/git -C add docs/probes/2026-09-16-skills-discovery +/usr/bin/git -C commit -m "Add the probe orchestrator with scenarios S0 to S4 (#961)" -m "Co-Authored-By: Claude Fable 5.1 " +``` + +--- + +### Task 10: ACP one-turn driver + +**Files:** +- Create: `docs/probes/2026-09-16-skills-discovery/lib/acp_driver.py` +- Create: `docs/probes/2026-09-16-skills-discovery/selftest_servers.py` +- Modify: `docs/probes/2026-09-16-skills-discovery/selftest.py` + +**Interfaces:** +- Consumes: `AcpClient` from `docs/probes/2026-08-04-acp-reconnect-c0/acp_c0_probe.py` (its `start()` merges `extra_env` over `os.environ`, so the subclass below replaces `start()` to use the sandbox environment verbatim), `harness.base.AskResult`. +- Produces: `acp_ask(argv: list[str], cwd: Path, env: dict, prompt: str, stderr_path: Path, timeout: float = 180.0) -> AskResult`. Sequence: `initialize` → `session/new {cwd, mcpServers: []}` → `session/prompt {sessionId, prompt:[{type:"text", text}]}`. `reply_text` is the concatenation of `session/update` notifications whose `update.sessionUpdate` is `agent_message_chunk` or `agent_message` (`update.content.text` or each `content[i].text`). `raw` is the JSON frame log. `notes` carries `stopReason=` and any error. `first_request_at` is stamped just before `session/prompt`. +- Produces: `selftest_servers.py`, a stdlib script that fakes a vendor: `python3 selftest_servers.py acp|appserver|pirpc`. Every fake reads `.fake/skills/kcap-probe-*/SKILL.md` under its cwd and answers each prompt with `=PROBE-BODY-` lines or `NO-SKILL`. + +- [ ] **Step 1: Write the fake servers** + +`selftest_servers.py`: + +```python +#!/usr/bin/env python3 +"""Fake vendor processes for the kit's self-tests: acp, appserver (codex app-server) and pirpc.""" +from __future__ import annotations + +import json +import re +import sys +from pathlib import Path + +TOKEN_RE = re.compile(r"PROBE-BODY-([0-9a-f]{12})") + + +def answer() -> str: + lines = [] + for f in sorted(Path.cwd().glob(".fake/skills/kcap-probe-*/SKILL.md")): + m = TOKEN_RE.search(f.read_text()) + if m: + lines.append(f"{f.parent.name}=PROBE-BODY-{m.group(1)}") + return "\n".join(lines) if lines else "NO-SKILL" + + +def send(obj: dict) -> None: + sys.stdout.write(json.dumps(obj) + "\n") + sys.stdout.flush() + + +def acp() -> None: + for line in sys.stdin: + msg = json.loads(line) + m, i, p = msg.get("method"), msg.get("id"), msg.get("params") or {} + if m == "initialize": + send({"jsonrpc": "2.0", "id": i, "result": {"protocolVersion": 1, "agentCapabilities": {}}}) + elif m == "session/new": + send({"jsonrpc": "2.0", "id": i, "result": {"sessionId": "s1"}}) + elif m == "session/prompt": + send({"jsonrpc": "2.0", "method": "session/update", "params": { + "sessionId": p["sessionId"], + "update": {"sessionUpdate": "agent_message_chunk", "content": {"type": "text", "text": answer()}}}}) + send({"jsonrpc": "2.0", "id": i, "result": {"stopReason": "end_turn"}}) + elif i is not None: + send({"jsonrpc": "2.0", "id": i, "error": {"code": -32601, "message": "unknown"}}) + + +def appserver() -> None: + for line in sys.stdin: + msg = json.loads(line) + m, i, p = msg.get("method"), msg.get("id"), msg.get("params") or {} + if m == "initialize": + send({"jsonrpc": "2.0", "id": i, "result": {}}) + elif m == "hooks/list": + send({"jsonrpc": "2.0", "id": i, "result": {"hooks": []}}) + elif m == "thread/start": + send({"jsonrpc": "2.0", "id": i, "result": {"thread": {"id": "t1"}, "model": "fake"}}) + elif m == "turn/start": + send({"jsonrpc": "2.0", "id": i, "result": {"turn": {"id": "u1"}}}) + send({"jsonrpc": "2.0", "method": "item/completed", "params": { + "item": {"type": "agentMessage", "id": "m1", "text": answer()}}}) + send({"jsonrpc": "2.0", "method": "turn/completed", "params": {"turn": {"id": "u1", "status": "completed"}}}) + elif i is not None: + send({"jsonrpc": "2.0", "id": i, "error": {"code": -32601, "message": "unknown"}}) + + +def pirpc() -> None: + for line in sys.stdin: + msg = json.loads(line) + if msg.get("type") == "prompt": + send({"id": msg.get("id"), "type": "response", "success": True}) + send({"type": "agent_start"}) + send({"type": "message_end", "message": {"role": "assistant", "content": [{"type": "text", "text": answer()}]}}) + send({"type": "agent_settled"}) + + +if __name__ == "__main__": + {"acp": acp, "appserver": appserver, "app-server": appserver, "pirpc": pirpc}[sys.argv[1]]() +``` + +Make it executable so a driver can launch it exactly like a vendor binary: `chmod 755 $KIT/selftest_servers.py`. The `app-server` alias lets `appserver_ask` run it as ` app-server`. + +- [ ] **Step 2: Write the failing test** + +Append to `selftest.py`: + +```python +from lib.acp_driver import acp_ask # noqa: E402 + +SERVERS = KIT / "selftest_servers.py" + + +def _fake_repo(d: str, skill: ProbeSkill) -> Path: + repo = Path(d) / "repo" + write_skill(repo / ".fake" / "skills", skill) + return repo + + +class AcpDriverTests(unittest.TestCase): + def test_one_turn(self): + with tempfile.TemporaryDirectory() as d: + skill = ProbeSkill.fresh() + repo = _fake_repo(d, skill) + res = acp_ask([sys.executable, str(SERVERS), "acp"], repo, dict(os.environ), single_prompt(skill), + Path(d) / "acp.stderr.log", timeout=30) + self.assertIn(skill.body_token, res.reply_text) + self.assertIn("stopReason=end_turn", res.notes) + self.assertGreaterEqual(res.first_request_at, res.started_at) + self.assertEqual(res.argv[-1], "acp") +``` + +- [ ] **Step 3: Run to verify failure** + +Run: `python3 $KIT/selftest.py -v` +Expected: `ModuleNotFoundError: No module named 'lib.acp_driver'` + +- [ ] **Step 4: Implement** + +`lib/acp_driver.py`: + +```python +from __future__ import annotations + +import asyncio +import json +import sys +import time +from pathlib import Path + +from harness.base import AskResult + +KIT = Path(__file__).resolve().parent.parent +sys.path.insert(0, str(KIT.parent / "2026-08-04-acp-reconnect-c0")) +from acp_c0_probe import AcpClient # noqa: E402 + +INIT_PARAMS = { + "protocolVersion": 1, + "clientCapabilities": {"fs": {"readTextFile": False, "writeTextFile": False}, "terminal": False}, +} + + +class IsolatedAcpClient(AcpClient): + """AcpClient whose child sees exactly the sandbox environment, not the developer's.""" + + def __init__(self, argv, cwd, env, stderr_path): + super().__init__(argv, cwd, frames=[], phase_ref=["turn"], label="acp", stderr_path=str(stderr_path)) + self.env = env + + async def start(self): + stderr_f = open(self.stderr_path, "ab") + self.proc = await asyncio.create_subprocess_exec( + *self.argv, cwd=self.cwd, env=self.env, + stdin=asyncio.subprocess.PIPE, stdout=asyncio.subprocess.PIPE, stderr=stderr_f) + self.reader_task = asyncio.create_task(self._read_loop()) + self.record("mark", {"event": "spawned", "pid": self.proc.pid, "argv": self.argv}) + + +def agent_text(frames: list[dict]) -> str: + out = [] + for f in frames: + fr = f.get("frame") or {} + if fr.get("method") != "session/update": + continue + upd = ((fr.get("params") or {}).get("update") or {}) + if upd.get("sessionUpdate") not in ("agent_message_chunk", "agent_message"): + continue + content = upd.get("content") + if isinstance(content, dict) and isinstance(content.get("text"), str): + out.append(content["text"]) + elif isinstance(content, list): + out.extend(c["text"] for c in content if isinstance(c, dict) and isinstance(c.get("text"), str)) + return "".join(out) + + +async def _turn(argv, cwd, env, prompt, stderr_path, timeout) -> AskResult: + client = IsolatedAcpClient(argv, str(cwd), env, stderr_path) + started = time.time() + first = started + text, notes = "", [] + await client.start() + try: + await client.request("initialize", INIT_PARAMS, timeout=90) + new = await client.request("session/new", {"cwd": str(cwd), "mcpServers": []}, timeout=120) + sid = (new.get("result") or {}).get("sessionId") + if not sid: + notes.append(f"session/new failed: {json.dumps(new)[:500]}") + else: + first = time.time() + resp = await client.request( + "session/prompt", {"sessionId": sid, "prompt": [{"type": "text", "text": prompt}]}, timeout=timeout) + notes.append(f"stopReason={(resp.get('result') or {}).get('stopReason')}") + if "error" in resp: + notes.append(f"error={json.dumps(resp['error'])[:500]}") + text = agent_text(client.frames) + except Exception as ex: # noqa: BLE001 + notes.append(f"exception={ex!r}") + finally: + await client.shutdown() + return AskResult(reply_text=text, raw=json.dumps(client.frames), argv=list(argv), started_at=started, + first_request_at=first, stderr_path=str(stderr_path), exit_code=client.proc.returncode, + notes=" ".join(notes)) + + +def acp_ask(argv: list[str], cwd: Path, env: dict, prompt: str, stderr_path: Path, timeout: float = 180.0) -> AskResult: + return asyncio.run(_turn(argv, cwd, env, prompt, stderr_path, timeout)) +``` + +- [ ] **Step 5: Run to verify pass** + +Run: `python3 $KIT/selftest.py -v` +Expected: 29 tests, `OK` + +- [ ] **Step 6: Commit** + +``` +/usr/bin/git -C add docs/probes/2026-09-16-skills-discovery +/usr/bin/git -C commit -m "Add the ACP one-turn driver and fake vendor servers (#961)" -m "Co-Authored-By: Claude Fable 5.1 " +``` + +--- + +### Task 11: Codex app-server and Pi RPC drivers + +**Files:** +- Create: `docs/probes/2026-09-16-skills-discovery/lib/jsonl_child.py` +- Create: `docs/probes/2026-09-16-skills-discovery/lib/appserver_driver.py` +- Create: `docs/probes/2026-09-16-skills-discovery/lib/pirpc_driver.py` +- Modify: `docs/probes/2026-09-16-skills-discovery/selftest.py` + +**Interfaces:** +- Produces (`lib/jsonl_child.py`): `class JsonlChild(argv, cwd, env, stderr_path)` with `start()`, `send(obj)`, `recv(timeout) -> dict | None` (next JSON line, `None` on EOF), `close_stdin()`, `stop()`, `frames: list[dict]` (everything sent and received), `returncode`. Synchronous, thread-backed reader with a queue. +- Produces (`lib/appserver_driver.py`): `appserver_ask(binary: str, cwd: Path, env: dict, prompt: str, stderr_path: Path, timeout: float = 180.0, extra_argv: list[str] = ()) -> AskResult`. Sequence: spawn `[binary, "app-server", *extra_argv]`; `initialize {clientInfo:{name:"kcap-probe",version:"1"}, capabilities:{}}`; `hooks/list {}`; when any hook in `result.hooks[]` has `trustStatus != "trusted"` and a `currentHash`, stop the child and respawn with `-c hooks.state={""={trusted_hash=""},…}` appended (the same full-table override the daemon uses; dotted keys silently fail); `thread/start {cwd, sandbox:"read-only", approvalPolicy:"never", approvalsReviewer:"user"}` → `result.thread.id`; `turn/start {threadId, input:[{type:"text",text}], sandboxPolicy:{type:"readOnly"}, approvalPolicy:"never", approvalsReviewer:"user"}`; read notifications until `turn/completed`; `reply_text` is the `params.item.text` of `item/completed` items with `type == "agentMessage"`, or the accumulated `item/agentMessage/delta` `params.delta` values when no completed item arrived. Server requests are answered with `-32601`. `notes` records `hook_trust=` and the turn status. +- Produces (`lib/pirpc_driver.py`): `pirpc_ask(argv: list[str], cwd: Path, env: dict, prompt: str, stderr_path: Path, timeout: float = 180.0) -> AskResult`. Sends `{"id":"1","type":"prompt","message":,"streamingBehavior":"followUp"}`; reads until `{"type":"agent_settled"}`; `reply_text` joins `message.content[].text` of `message_end` events whose `message.role == "assistant"`. + +- [ ] **Step 1: Write the failing tests** + +Append to `selftest.py`: + +```python +from lib.appserver_driver import appserver_ask # noqa: E402 +from lib.pirpc_driver import pirpc_ask # noqa: E402 +from lib.jsonl_child import JsonlChild # noqa: E402 + + +class JsonlDriversTests(unittest.TestCase): + def test_jsonl_child_round_trip(self): + with tempfile.TemporaryDirectory() as d: + c = JsonlChild([sys.executable, str(SERVERS), "pirpc"], Path(d), dict(os.environ), Path(d) / "c.stderr.log") + c.start() + c.send({"id": "1", "type": "prompt", "message": "x", "streamingBehavior": "followUp"}) + first = c.recv(10) + self.assertEqual(first["type"], "response") + c.close_stdin() + c.stop() + self.assertGreaterEqual(len(c.frames), 2) + + def test_appserver_turn(self): + with tempfile.TemporaryDirectory() as d: + skill = ProbeSkill.fresh() + repo = _fake_repo(d, skill) + res = appserver_ask(str(SERVERS), repo, dict(os.environ), single_prompt(skill), + Path(d) / "as.stderr.log", timeout=30) + self.assertIn(skill.body_token, res.reply_text) + self.assertIn("hook_trust=none", res.notes) + self.assertEqual(res.argv[:2], [str(SERVERS), "app-server"]) + + def test_pirpc_turn(self): + with tempfile.TemporaryDirectory() as d: + skill = ProbeSkill.fresh() + repo = _fake_repo(d, skill) + res = pirpc_ask([sys.executable, str(SERVERS), "pirpc"], repo, dict(os.environ), single_prompt(skill), + Path(d) / "pi.stderr.log", timeout=30) + self.assertIn(skill.body_token, res.reply_text) +``` + +Note: the fake server treats the literal `app-server` argument that `appserver_ask` inserts as its mode selector; the assertion on `argv[:2]` pins that insertion. + +- [ ] **Step 2: Run to verify failure** + +Run: `python3 $KIT/selftest.py -v` +Expected: `ModuleNotFoundError: No module named 'lib.appserver_driver'` + +- [ ] **Step 3: Implement the child** + +`lib/jsonl_child.py`: + +```python +from __future__ import annotations + +import json +import queue +import subprocess +import threading +import time +from pathlib import Path + + +class JsonlChild: + def __init__(self, argv: list[str], cwd: Path, env: dict, stderr_path: Path) -> None: + self.argv = list(argv) + self.cwd = str(cwd) + self.env = env + self.stderr_path = stderr_path + self.frames: list[dict] = [] + self._q: queue.Queue = queue.Queue() + self.proc: subprocess.Popen | None = None + + def start(self) -> None: + self._err = open(self.stderr_path, "ab") + self.proc = subprocess.Popen(self.argv, cwd=self.cwd, env=self.env, stdin=subprocess.PIPE, + stdout=subprocess.PIPE, stderr=self._err, text=True, bufsize=1) + threading.Thread(target=self._reader, daemon=True).start() + + def _reader(self) -> None: + assert self.proc and self.proc.stdout + for line in self.proc.stdout: + line = line.strip() + if not line: + continue + try: + obj = json.loads(line) + except json.JSONDecodeError: + obj = {"unparseable": line[:2000]} + self.frames.append({"t": time.time(), "dir": "in", "frame": obj}) + self._q.put(obj) + self._q.put(None) + + def send(self, obj: dict) -> None: + assert self.proc and self.proc.stdin + self.frames.append({"t": time.time(), "dir": "out", "frame": obj}) + self.proc.stdin.write(json.dumps(obj) + "\n") + self.proc.stdin.flush() + + def recv(self, timeout: float) -> dict | None: + try: + return self._q.get(timeout=timeout) + except queue.Empty: + raise TimeoutError(f"no frame within {timeout}s") + + def close_stdin(self) -> None: + if self.proc and self.proc.stdin: + try: + self.proc.stdin.close() + except OSError: + pass + + def stop(self, grace: float = 5.0) -> None: + if not self.proc: + return + self.close_stdin() + try: + self.proc.wait(timeout=grace) + except subprocess.TimeoutExpired: + self.proc.kill() + self.proc.wait() + self._err.close() + + @property + def returncode(self) -> int | None: + return self.proc.returncode if self.proc else None +``` + +- [ ] **Step 4: Implement the app-server driver** + +`lib/appserver_driver.py`: + +```python +from __future__ import annotations + +import json +import time +from pathlib import Path + +from harness.base import AskResult +from lib.jsonl_child import JsonlChild + + +class _Rpc: + def __init__(self, child: JsonlChild) -> None: + self.child = child + self.next_id = 0 + self.notifications: list[dict] = [] + + def request(self, method: str, params: dict, timeout: float) -> dict: + self.next_id += 1 + rid = self.next_id + self.child.send({"jsonrpc": "2.0", "id": rid, "method": method, "params": params}) + deadline = time.time() + timeout + while True: + msg = self.child.recv(max(0.1, deadline - time.time())) + if msg is None: + raise ConnectionError(f"EOF waiting for {method}") + if msg.get("id") == rid and "method" not in msg: + return msg + self._absorb(msg) + + def _absorb(self, msg: dict) -> None: + if "method" in msg and "id" in msg: + self.child.send({"jsonrpc": "2.0", "id": msg["id"], "error": {"code": -32601, "message": "unsupported"}}) + elif "method" in msg: + self.notifications.append(msg) + + def wait_notification(self, method: str, timeout: float) -> dict | None: + deadline = time.time() + timeout + while time.time() < deadline: + msg = self.child.recv(max(0.1, deadline - time.time())) + if msg is None: + return None + self._absorb(msg) + if msg.get("method") == method: + return msg + return None + + +def _toml_string(s: str) -> str: + return json.dumps(s) + + +def hook_state_override(hooks: list[dict]) -> str | None: + untrusted = [h for h in hooks if h.get("trustStatus") != "trusted" and h.get("currentHash") and h.get("key")] + if not untrusted: + return None + entries = ",".join(f"{_toml_string(h['key'])}={{trusted_hash={_toml_string(h['currentHash'])}}}" for h in untrusted) + return f"hooks.state={{{entries}}}" + + +def appserver_ask(binary: str, cwd: Path, env: dict, prompt: str, stderr_path: Path, + timeout: float = 180.0, extra_argv: list[str] = ()) -> AskResult: + started = time.time() + argv = [binary, "app-server", *extra_argv] + notes = [] + text = "" + first = started + child = JsonlChild(argv, cwd, env, stderr_path) + child.start() + rpc = _Rpc(child) + try: + init = {"clientInfo": {"name": "kcap-probe", "version": "1"}, "capabilities": {}} + rpc.request("initialize", init, 60) + hooks = (rpc.request("hooks/list", {}, 60).get("result") or {}).get("hooks") or [] + override = hook_state_override(hooks) + if override: + child.stop() + argv = [*argv, "-c", override] + child = JsonlChild(argv, cwd, env, stderr_path) + child.start() + rpc = _Rpc(child) + rpc.request("initialize", init, 60) + notes.append("hook_trust=seeded") + else: + notes.append("hook_trust=trusted" if hooks else "hook_trust=none") + thread = rpc.request("thread/start", { + "cwd": str(cwd), "sandbox": "read-only", "approvalPolicy": "never", "approvalsReviewer": "user", + }, 120) + tid = ((thread.get("result") or {}).get("thread") or {}).get("id") + if not tid: + notes.append(f"thread/start failed: {json.dumps(thread)[:500]}") + else: + first = time.time() + rpc.request("turn/start", { + "threadId": tid, "input": [{"type": "text", "text": prompt}], + "sandboxPolicy": {"type": "readOnly"}, "approvalPolicy": "never", "approvalsReviewer": "user", + }, 60) + done = rpc.wait_notification("turn/completed", timeout) + notes.append(f"turn={(((done or {}).get('params') or {}).get('turn') or {}).get('status')}") + completed = [n["params"]["item"]["text"] for n in rpc.notifications + if n.get("method") == "item/completed" + and (n.get("params") or {}).get("item", {}).get("type") == "agentMessage" + and isinstance(n["params"]["item"].get("text"), str)] + deltas = [n["params"].get("delta", "") for n in rpc.notifications if n.get("method") == "item/agentMessage/delta"] + text = "\n".join(completed) if completed else "".join(deltas) + except Exception as ex: # noqa: BLE001 + notes.append(f"exception={ex!r}") + finally: + child.stop() + return AskResult(reply_text=text, raw=json.dumps(child.frames), argv=argv, started_at=started, + first_request_at=first, stderr_path=str(stderr_path), exit_code=child.returncode, + notes=" ".join(notes)) +``` + +- [ ] **Step 5: Implement the Pi RPC driver** + +`lib/pirpc_driver.py`: + +```python +from __future__ import annotations + +import json +import time +from pathlib import Path + +from harness.base import AskResult +from lib.jsonl_child import JsonlChild + + +def pirpc_ask(argv: list[str], cwd: Path, env: dict, prompt: str, stderr_path: Path, timeout: float = 180.0) -> AskResult: + started = time.time() + child = JsonlChild(argv, cwd, env, stderr_path) + child.start() + texts, notes = [], [] + first = time.time() + try: + child.send({"id": "1", "type": "prompt", "message": prompt, "streamingBehavior": "followUp"}) + deadline = time.time() + timeout + while time.time() < deadline: + msg = child.recv(max(0.1, deadline - time.time())) + if msg is None: + notes.append("eof before agent_settled") + break + t = msg.get("type") + if t == "message_end" and (msg.get("message") or {}).get("role") == "assistant": + for part in (msg["message"].get("content") or []): + if isinstance(part, dict) and part.get("type") == "text": + texts.append(part.get("text", "")) + if msg["message"].get("stopReason") == "error": + notes.append(f"error={msg['message'].get('errorMessage')}") + elif t == "response" and msg.get("success") is False: + notes.append(f"prompt rejected: {json.dumps(msg)[:300]}") + elif t == "extension_ui_request": + notes.append(f"blocked on ui request {msg.get('method')}") + break + elif t == "agent_settled": + break + else: + notes.append("timeout") + except Exception as ex: # noqa: BLE001 + notes.append(f"exception={ex!r}") + finally: + child.stop() + return AskResult(reply_text="\n".join(texts), raw=json.dumps(child.frames), argv=list(argv), started_at=started, + first_request_at=first, stderr_path=str(stderr_path), exit_code=child.returncode, + notes=" ".join(notes)) +``` + +- [ ] **Step 6: Run to verify pass** + +Run: `python3 $KIT/selftest.py -v` +Expected: 32 tests, `OK` + +- [ ] **Step 7: Commit** + +``` +/usr/bin/git -C add docs/probes/2026-09-16-skills-discovery +/usr/bin/git -C commit -m "Add the Codex app-server and Pi RPC drivers (#961)" -m "Co-Authored-By: Claude Fable 5.1 " +``` + +--- + +### Task 12: Claude adapter + +**Files:** +- Create: `docs/probes/2026-09-16-skills-discovery/harness/claude.py` +- Modify: `docs/probes/2026-09-16-skills-discovery/harness/__init__.py` +- Modify: `docs/probes/2026-09-16-skills-discovery/selftest.py` + +**Interfaces:** +- Consumes: `Adapter`, `HookInfo`, `print_ask`. +- Produces: `class ClaudeAdapter(Adapter)`, entry `claude`, lever `CLAUDE_CONFIG_DIR`, native root `.claude/skills`, documented roots `{".claude/skills"}`, modes `("print",)` (the daemon launches Claude the same way). Hook goes to `/settings.json` under `hooks.SessionStart`, the location kcap's plugin hooks resolve to. Print argv: `claude -p --output-format json --max-turns 4 --strict-mcp-config --allowedTools Skill --setting-sources user`; reply is the top-level `result` field. Environment switch `KCAP_PROBE_CLAUDE_REAL_CONFIG=1` makes the adapter use the developer's real config root (for a keychain-only credential) with hooks supplied through `--settings ` and `KCAP_SKIP=1` so kcap's own hooks stand down. + +- [ ] **Step 1: Write the failing test** + +Append to `selftest.py` a generic hook-file test that every adapter task extends by adding its class to `ENTRIES`: + +```python +from harness import ENTRIES # noqa: E402 + + +class AdapterHookFilesTests(unittest.TestCase): + def test_every_adapter_writes_a_hook_referencing_the_script(self): + for name, cls in ENTRIES.items(): + if name == "fake": + continue + with self.subTest(entry=name), tempfile.TemporaryDirectory() as d: + a = cls() + sb = new_sandbox(a.lever, None, [], base=Path(d)) + a.prepare(sb) + script = sb.config_root / "probe-hook.sh" + script.write_text("#!/bin/sh\nexit 0\n") + info = a.install_startup_hook(sb, script) + self.assertTrue(Path(info.config_path).exists(), info) + self.assertIn(str(script), Path(info.config_path).read_text()) + self.assertTrue(info.mechanism) + self.assertTrue(a.native_root in a.documented_roots, name) +``` + +Also append: + +```python +class ClaudeAdapterTests(unittest.TestCase): + def test_prepare_trusts_repo(self): + from harness.claude import ClaudeAdapter + with tempfile.TemporaryDirectory() as d: + a = ClaudeAdapter() + sb = new_sandbox(a.lever, None, [], base=Path(d)) + a.prepare(sb) + cfg = json.loads((sb.config_root / ".claude.json").read_text()) + self.assertTrue(cfg["projects"][str(sb.repo)]["hasTrustDialogAccepted"]) + self.assertEqual(a.print_argv(sb, "hi")[:3], [a.binary_path() or "claude", "-p", "hi"]) +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `python3 $KIT/selftest.py -v` +Expected: `ModuleNotFoundError: No module named 'harness.claude'` + +- [ ] **Step 3: Implement** + +`harness/claude.py`: + +```python +from __future__ import annotations + +import json +import os +import subprocess +from pathlib import Path + +from harness.base import Adapter, AskResult, HookInfo +from lib.isolation import Sandbox +from lib.print_driver import print_ask + + +class ClaudeAdapter(Adapter): + entry = "claude" + harness = "claude" + binary = "claude" + lever = "CLAUDE_CONFIG_DIR" + credential_files = [".credentials.json"] + native_root = ".claude/skills" + documented_roots = frozenset({".claude/skills"}) + modes = ("print",) + + def __init__(self) -> None: + self.real_config = os.environ.get("KCAP_PROBE_CLAUDE_REAL_CONFIG") == "1" + + def real_root(self) -> Path | None: + return Path(os.environ.get("CLAUDE_CONFIG_DIR") or (Path.home() / ".claude")) + + def prepare(self, sb: Sandbox) -> None: + if self.real_config: + sb.env["CLAUDE_CONFIG_DIR"] = str(self.real_root()) + sb.env["KCAP_SKIP"] = "1" + (sb.config_root / "settings.json").write_text("{}\n") + return + (sb.config_root / ".claude.json").write_text(json.dumps({ + "hasCompletedOnboarding": True, + "projects": {str(sb.repo): {"hasTrustDialogAccepted": True}}, + }, indent=2)) + settings = sb.config_root / "settings.json" + if not settings.exists(): + settings.write_text("{}\n") + + def check_auth(self, sb: Sandbox) -> bool | None: + out = subprocess.run([self.binary_path() or self.binary, "auth", "status"], env=sb.env, + capture_output=True, text=True, timeout=60) + try: + return bool(json.loads(out.stdout).get("loggedIn")) + except (json.JSONDecodeError, AttributeError): + return False + + def install_startup_hook(self, sb: Sandbox, script: Path) -> HookInfo: + settings = sb.config_root / "settings.json" + data = json.loads(settings.read_text() or "{}") + data.setdefault("hooks", {})["SessionStart"] = [ + {"hooks": [{"type": "command", "command": str(script), "timeout": 10}]} + ] + settings.write_text(json.dumps(data, indent=2) + "\n") + where = "--settings file" if self.real_config else "settings.json" + return HookInfo(mechanism=f"{where} hooks.SessionStart", config_path=str(settings)) + + def print_argv(self, sb: Sandbox, prompt: str) -> list[str]: + argv = [self.binary_path() or self.binary, "-p", prompt, "--output-format", "json", "--max-turns", "4", + "--strict-mcp-config", "--allowedTools", "Skill"] + if self.real_config: + argv += ["--settings", str(sb.config_root / "settings.json")] + else: + argv += ["--setting-sources", "user"] + return argv + + def ask(self, sb: Sandbox, mode: str, prompt: str) -> AskResult: + def extract(raw: str) -> str: + obj = json.loads(raw) + if obj.get("is_error"): + raise ValueError(obj) + return obj.get("result") or "" + return print_ask(self.print_argv(sb, prompt), sb.repo, sb.env, sb.root / "claude.stderr.log", + self.turn_timeout, extract=extract) +``` + +Register in `harness/__init__.py`: + +```python +from harness.claude import ClaudeAdapter +ENTRIES["claude"] = ClaudeAdapter +``` + +(`ENTRIES` becomes a dict literal followed by assignments; keep `fake` first.) + +- [ ] **Step 4: Run to verify pass** + +Run: `python3 $KIT/selftest.py -v` +Expected: 34 tests, `OK` + +- [ ] **Step 5: Free phase against the real binary** + +Run: `python3 $KIT/probe.py --harness claude --mode print` +Expected: `claude: 2.1.273 (Claude Code) auth_ok=True`. If `auth_ok=False` (keychain credential not reachable under the isolated root), rerun with `KCAP_PROBE_CLAUDE_REAL_CONFIG=1` and keep that variable exported for every later Claude run; record the lever actually used in `findings.md`. + +- [ ] **Step 6: Positive control, one turn** + +Run: `python3 $KIT/probe.py --harness claude --mode print --scenario S0 --scenario S1 --turn --runs 1` +Expected: `claude print S0/none ... -> not_visible` and `claude print S1/native ... -> visible_first_turn`. If S1 is `not_visible`, open `out/claude/print/S1/S1_native/run1.json`, read `reply` and the stderr log, adjust `print_argv` (the usual culprits are a trust prompt or the `Skill` tool name) and rerun; note the change in `findings.md`. + +- [ ] **Step 7: Commit** + +``` +/usr/bin/git -C add docs/probes/2026-09-16-skills-discovery +/usr/bin/git -C commit -m "Add the Claude probe adapter (#961)" -m "Co-Authored-By: Claude Fable 5.1 " +``` + +--- + +### Task 13: Codex adapter + +**Files:** +- Create: `docs/probes/2026-09-16-skills-discovery/harness/codex.py` +- Modify: `docs/probes/2026-09-16-skills-discovery/harness/__init__.py` + +**Interfaces:** +- Consumes: `print_ask`, `appserver_ask`. +- Produces: `class CodexAdapter(Adapter)`, entry `codex`, lever `CODEX_HOME`, credential `auth.json`, native root `.agents/skills`, documented roots `{".agents/skills"}`. Hook file `/hooks.json` in kcap's shape. Print mode: `codex exec --skip-git-repo-check --sandbox read-only --color never --dangerously-bypass-hook-trust --output-last-message -` with the prompt on stdin; reply is the file. Daemon mode: `appserver_ask` (which seeds hook trust itself). + +- [ ] **Step 1: Implement** + +`harness/codex.py`: + +```python +from __future__ import annotations + +import json +import subprocess +from pathlib import Path + +from harness.base import Adapter, AskResult, HookInfo +from lib.appserver_driver import appserver_ask +from lib.isolation import Sandbox +from lib.print_driver import print_ask + + +class CodexAdapter(Adapter): + entry = "codex" + harness = "codex" + binary = "codex" + lever = "CODEX_HOME" + credential_files = ["auth.json"] + native_root = ".agents/skills" + documented_roots = frozenset({".agents/skills"}) + + def real_root(self) -> Path | None: + return Path.home() / ".codex" + + def prepare(self, sb: Sandbox) -> None: + (sb.config_root / "config.toml").write_text( + f'[projects.{json.dumps(str(sb.repo))}]\ntrust_level = "trusted"\n') + + def check_auth(self, sb: Sandbox) -> bool | None: + out = subprocess.run([self.binary_path() or self.binary, "login", "status"], env=sb.env, + capture_output=True, text=True, timeout=60) + return out.returncode == 0 and "Logged in" in (out.stdout + out.stderr) + + def install_startup_hook(self, sb: Sandbox, script: Path) -> HookInfo: + hooks = sb.config_root / "hooks.json" + hooks.write_text(json.dumps({"hooks": {"SessionStart": [ + {"hooks": [{"type": "command", "command": str(script), "timeout": 30}]} + ]}}, indent=2) + "\n") + return HookInfo(mechanism="hooks.json SessionStart", config_path=str(hooks)) + + def ask(self, sb: Sandbox, mode: str, prompt: str) -> AskResult: + binary = self.binary_path() or self.binary + if mode == "daemon": + return appserver_ask(binary, sb.repo, sb.env, prompt, sb.root / "codex-appserver.stderr.log", self.turn_timeout) + last = sb.root / "last-message.txt" + argv = [binary, "exec", "--skip-git-repo-check", "--sandbox", "read-only", "--color", "never", + "--dangerously-bypass-hook-trust", "--output-last-message", str(last), "-"] + res = print_ask(argv, sb.repo, sb.env, sb.root / "codex.stderr.log", self.turn_timeout, stdin_text=prompt) + if last.exists(): + res.reply_text = last.read_text().strip() + return res +``` + +Register: `from harness.codex import CodexAdapter` and `ENTRIES["codex"] = CodexAdapter`. + +- [ ] **Step 2: Self-tests still pass** + +Run: `python3 $KIT/selftest.py -v` +Expected: 34 tests, `OK` (the hook-files test now covers `codex`). + +- [ ] **Step 3: Free phase and positive control in both modes** + +Run: `python3 $KIT/probe.py --harness codex --mode print` then `python3 $KIT/probe.py --harness codex --mode daemon` +Expected: `codex: codex-cli 0.154.0 auth_ok=True` twice. + +Run: `python3 $KIT/probe.py --harness codex --mode print --scenario S0 --scenario S1 --turn --runs 1` and the same with `--mode daemon` +Expected: S0 `not_visible`, S1 `visible_first_turn` in both modes. If daemon mode reports `hook_trust=` anything but `none` on S1 (no hook installed yet) the `hooks/list` shape differs from the daemon's; compare against `src/Capacitor.Cli.Daemon/Harness/Codex/CodexAppServerHostedAgentRuntime.cs` field names (`key`, `trustStatus`, `currentHash`) and fix `hook_state_override`. + +- [ ] **Step 4: Commit** + +``` +/usr/bin/git -C add docs/probes/2026-09-16-skills-discovery +/usr/bin/git -C commit -m "Add the Codex probe adapter (#961)" -m "Co-Authored-By: Claude Fable 5.1 " +``` + +--- + +### Task 14: Gemini adapter + +**Files:** +- Create: `docs/probes/2026-09-16-skills-discovery/harness/gemini.py` +- Modify: `docs/probes/2026-09-16-skills-discovery/harness/__init__.py` + +**Interfaces:** +- Produces: `class GeminiAdapter(Adapter)`, entry `gemini`, lever `GEMINI_CLI_HOME` (the parent of `.gemini`), credentials `.gemini/oauth_creds.json`, `.gemini/google_accounts.json`, `.gemini/installation_id` copied from `$HOME`, native root `.gemini/skills`, documented roots `{".gemini/skills", ".agents/skills"}`. Hook in `/.gemini/settings.json` under `hooks.SessionStart` (kcap's shape, timeout in ms). Print: `gemini -p -o json --approval-mode yolo`, reply `response`. Daemon: `gemini --experimental-acp --skip-trust --approval-mode yolo` over `acp_ask`. Catalogue: `gemini skills list`. + +- [ ] **Step 1: Update Gemini first** + +Run: `npm install -g @google/gemini-cli@latest` then `gemini --version` +Expected: `0.60.0` (the version published at planning time; record whatever prints). + +- [ ] **Step 2: Implement** + +`harness/gemini.py`: + +```python +from __future__ import annotations + +import json +import subprocess +from pathlib import Path + +from harness.base import Adapter, AskResult, HookInfo +from lib.acp_driver import acp_ask +from lib.isolation import Sandbox +from lib.print_driver import print_ask + + +class GeminiAdapter(Adapter): + entry = "gemini" + harness = "gemini" + binary = "gemini" + lever = "GEMINI_CLI_HOME" + credential_files = [".gemini/oauth_creds.json", ".gemini/google_accounts.json", ".gemini/installation_id"] + native_root = ".gemini/skills" + documented_roots = frozenset({".gemini/skills", ".agents/skills"}) + + def real_root(self) -> Path | None: + return Path.home() + + def _settings(self, sb: Sandbox) -> Path: + return sb.config_root / ".gemini" / "settings.json" + + def prepare(self, sb: Sandbox) -> None: + self._settings(sb).parent.mkdir(parents=True, exist_ok=True) + if not self._settings(sb).exists(): + self._settings(sb).write_text(json.dumps({"security": {"folderTrust": {"enabled": False}}}, indent=2) + "\n") + (sb.config_root / ".gemini" / "trustedFolders.json").write_text(json.dumps({str(sb.repo): "TRUST_FOLDER"}) + "\n") + + def check_auth(self, sb: Sandbox) -> bool | None: + return (sb.config_root / ".gemini" / "oauth_creds.json").exists() or None + + def install_startup_hook(self, sb: Sandbox, script: Path) -> HookInfo: + data = json.loads(self._settings(sb).read_text() or "{}") + data.setdefault("hooks", {})["SessionStart"] = [ + {"hooks": [{"name": "probe", "type": "command", "command": str(script), "timeout": 30000}]} + ] + self._settings(sb).write_text(json.dumps(data, indent=2) + "\n") + return HookInfo(mechanism="settings.json hooks.SessionStart", config_path=str(self._settings(sb))) + + def ask(self, sb: Sandbox, mode: str, prompt: str) -> AskResult: + binary = self.binary_path() or self.binary + if mode == "daemon": + return acp_ask([binary, "--experimental-acp", "--skip-trust", "--approval-mode", "yolo"], sb.repo, sb.env, + prompt, sb.root / "gemini-acp.stderr.log", self.turn_timeout) + argv = [binary, "-p", prompt, "-o", "json", "--approval-mode", "yolo"] + return print_ask(argv, sb.repo, sb.env, sb.root / "gemini.stderr.log", self.turn_timeout, + extract=lambda raw: json.loads(raw).get("response") or "") + + def list_catalogue(self, sb: Sandbox) -> str | None: + out = subprocess.run([self.binary_path() or self.binary, "skills", "list"], cwd=str(sb.repo), env=sb.env, + capture_output=True, text=True, timeout=120) + return out.stdout +``` + +Register `gemini`. + +- [ ] **Step 3: Self-tests, free phase, positive control** + +Run: `python3 $KIT/selftest.py -v` → `OK`. +Run: `python3 $KIT/probe.py --harness gemini --mode print` and `--mode daemon` → `auth_ok=True`. +Run: `python3 $KIT/probe.py --harness gemini --mode print --scenario S0 --scenario S1 --turn --runs 1` and the same for `--mode daemon`. +Expected: S1 `visible_first_turn`. If the reply mentions a consent or trust prompt, the `folderTrust` key or the skill activation consent is the cause: check `gemini --help` for the current approval flags and `docs/probes/2026-09-16-skills-discovery/out/gemini/print/S1/S1_native/gemini.stderr.log`; record the fix. + +- [ ] **Step 4: Commit** + +``` +/usr/bin/git -C add docs/probes/2026-09-16-skills-discovery +/usr/bin/git -C commit -m "Add the Gemini probe adapter (#961)" -m "Co-Authored-By: Claude Fable 5.1 " +``` + +--- + +### Task 15: Pi adapter + +**Files:** +- Create: `docs/probes/2026-09-16-skills-discovery/harness/pi.py` +- Modify: `docs/probes/2026-09-16-skills-discovery/harness/__init__.py` + +**Interfaces:** +- Produces: `class PiAdapter(Adapter)`, entry `pi`, lever `PI_CODING_AGENT_DIR` (the agent-state leaf), credentials `auth.json`, `models.json`, `settings.json` from `~/.pi/agent`, native root `.pi/skills`, documented roots `{".pi/skills", ".agents/skills"}`. Startup hook: extension `/extensions/probe.ts` whose `session_start` handler runs the script. Registration arm: extension `probe-register.ts` whose `session_start` writes the skill and whose `resources_discover` returns `{skillPaths: []}`. Print: `pi -p --mode json --approve -- `, reply from `message_end` assistant text parts. Daemon: `pi --mode rpc --approve` over `pirpc_ask`. + +- [ ] **Step 1: Update Pi first** + +Run: `npm install -g @earendil-works/pi-coding-agent@latest` then `pi --version`; record the version. + +- [ ] **Step 2: Implement** + +`harness/pi.py`: + +```python +from __future__ import annotations + +import json +import subprocess +from pathlib import Path + +from harness.base import Adapter, AskResult, HookInfo +from lib.isolation import Sandbox +from lib.pirpc_driver import pirpc_ask +from lib.print_driver import print_ask + +HOOK_EXT = """import {{ execFileSync }} from "node:child_process"; + +export default function (pi: any) {{ + pi.on("session_start", async () => {{ + try {{ execFileSync({script}, {{ stdio: "ignore" }}); }} catch {{}} + }}); +}} +""" + +REGISTER_EXT = """import {{ execFileSync }} from "node:child_process"; + +export default function (pi: any) {{ + pi.on("session_start", async () => {{ + try {{ execFileSync({script}, {{ stdio: "ignore" }}); }} catch {{}} + }}); + pi.on("resources_discover", async () => {{ + return {{ skillPaths: [{root}] }}; + }}); +}} +""" + + +class PiAdapter(Adapter): + entry = "pi" + harness = "pi" + binary = "pi" + lever = "PI_CODING_AGENT_DIR" + credential_files = ["auth.json", "models.json", "settings.json"] + native_root = ".pi/skills" + documented_roots = frozenset({".pi/skills", ".agents/skills"}) + + def real_root(self) -> Path | None: + return Path.home() / ".pi" / "agent" + + def check_auth(self, sb: Sandbox) -> bool | None: + auth = sb.config_root / "auth.json" + if not auth.exists(): + return False + for provider in json.loads(auth.read_text()).keys(): + out = subprocess.run([self.binary_path() or self.binary, "auth", "check", "--provider", provider, "--json"], + env=sb.env, capture_output=True, text=True, timeout=60) + if out.returncode == 0: + return True + return False + + def _ext_dir(self, sb: Sandbox) -> Path: + d = sb.config_root / "extensions" + d.mkdir(parents=True, exist_ok=True) + return d + + def install_startup_hook(self, sb: Sandbox, script: Path) -> HookInfo: + ext = self._ext_dir(sb) / "probe.ts" + ext.write_text(HOOK_EXT.format(script=json.dumps(str(script)))) + return HookInfo(mechanism="extension session_start", config_path=str(ext)) + + def install_registration(self, sb: Sandbox, skill_file: Path, body: str) -> HookInfo | None: + from lib.hook_script import stamp_path, write_hook_script + script = write_hook_script(sb.config_root, skill_file, body, stamp_path(sb.config_root)) + root = skill_file.parent.parent + ext = self._ext_dir(sb) / "probe-register.ts" + ext.write_text(REGISTER_EXT.format(script=json.dumps(str(script)), root=json.dumps(str(root)))) + return HookInfo(mechanism="extension session_start + resources_discover skillPaths", config_path=str(ext)) + + def ask(self, sb: Sandbox, mode: str, prompt: str) -> AskResult: + binary = self.binary_path() or self.binary + if mode == "daemon": + return pirpc_ask([binary, "--mode", "rpc", "--approve"], sb.repo, sb.env, prompt, + sb.root / "pi-rpc.stderr.log", self.turn_timeout) + + def extract(raw: str) -> str: + texts = [] + for line in raw.splitlines(): + try: + msg = json.loads(line) + except json.JSONDecodeError: + continue + if msg.get("type") == "message_end" and (msg.get("message") or {}).get("role") == "assistant": + texts += [p.get("text", "") for p in msg["message"].get("content") or [] + if isinstance(p, dict) and p.get("type") == "text"] + return "\n".join(texts) + argv = [binary, "-p", "--mode", "json", "--approve", "--", prompt] + return print_ask(argv, sb.repo, sb.env, sb.root / "pi.stderr.log", self.turn_timeout, extract=extract) +``` + +Register `pi`. + +- [ ] **Step 3: Self-tests, free phase, positive control** + +Run: `python3 $KIT/selftest.py -v` → `OK`. +Run: `python3 $KIT/probe.py --harness pi --mode print` and `--mode daemon` → `auth_ok=True`. +Run: `python3 $KIT/probe.py --harness pi --mode print --scenario S0 --scenario S1 --turn --runs 1` and `--mode daemon`. +Expected: S1 `visible_first_turn`. If Pi refuses because no default model is configured in the copied `settings.json`, export `KCAP_PROBE_PI_MODEL=` and add `["--model", os.environ["KCAP_PROBE_PI_MODEL"]]` to both argv lists when set; record it. + +- [ ] **Step 4: Commit** + +``` +/usr/bin/git -C add docs/probes/2026-09-16-skills-discovery +/usr/bin/git -C commit -m "Add the Pi probe adapter (#961)" -m "Co-Authored-By: Claude Fable 5.1 " +``` + +--- + +### Task 16: Install the missing vendor CLIs + +**Files:** none in the repo; this task changes the developer machine and records versions for `findings.md`. + +- [ ] **Step 1: Install** + +``` +curl https://cursor.com/install -fsS | bash +npm install -g @github/copilot +brew install --cask kiro-cli +npm install -g opencode-ai +mkdir -p ~/.local/opencode-v2 && npm install -g --prefix ~/.local/opencode-v2 @opencode/cli +curl -fsSL https://antigravity.google/cli/install.sh | bash +``` + +`~/.local/bin` must be on `PATH` for `cursor-agent` and `agy`. OpenCode V2 stays out of `PATH`; the adapter reads its binary from `KCAP_OPENCODE_V2_PATH` (default `~/.local/opencode-v2/bin/opencode`). + +- [ ] **Step 2: Record versions** + +Run each: `cursor-agent --version`, `copilot --version`, `kiro-cli --version`, `opencode --version`, `~/.local/opencode-v2/bin/opencode --version`, `agy --version`. +Expected: each prints a version; the V2 binary reports a `2.x` version and the V1 binary a `1.x` one. Paste all six into a scratch note for `findings.md`. + +- [ ] **Step 3: Logins (developer runs these, they open a browser)** + +Ask the developer to run, in this order, and report which succeeded: + +``` +cursor-agent login +kiro-cli login +opencode auth login +agy +``` + +(`agy` starts the OAuth flow on first launch; `/logout` ends it.) Copilot needs no login when `gh auth token` yields an OAuth token; the adapter passes it as `COPILOT_GITHUB_TOKEN`. Kiro's headless mode may additionally need `KIRO_API_KEY`; if `kiro-cli chat --no-interactive` refuses after login, the developer exports that key or the Kiro turn arms stay `untested`. + +- [ ] **Step 4: Capture the flags the installed builds expose** + +Run: `cursor-agent --help`, `copilot --help`, `kiro-cli chat --help`, `opencode run --help`, `~/.local/opencode-v2/bin/opencode run --help`, `agy --help`. +Expected flags, to be confirmed and corrected in the adapters that follow: Cursor `-p/--print`, `--output-format json|text|stream-json`, `--trust`, `--force`; Copilot `-p/--prompt`, `--allow-all-tools`, `-s/--silent`; Kiro `--no-interactive`, `--trust-all-tools`, `--agent`, `--output-format stream-json`; OpenCode `run` with `--format json`; agy `-p`, `--output-format stream-json`, `--dangerously-skip-permissions`, `--print-timeout`. Where a flag is absent or renamed, use the build's name in the adapter and note it in `findings.md`. + +--- + +### Task 17: Cursor adapter + +**Files:** +- Create: `docs/probes/2026-09-16-skills-discovery/harness/cursor.py` +- Modify: `docs/probes/2026-09-16-skills-discovery/harness/__init__.py` + +**Interfaces:** +- Produces: `class CursorAdapter(Adapter)`, entry `cursor`, lever `HOME` (no narrower lever exists), native root `.cursor/skills`, documented roots `{".cursor/skills", ".agents/skills", ".claude/skills", ".codex/skills"}`. Hook: `/.cursor/hooks.json` `{"version":1,"hooks":{"sessionStart":[{"command":