Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
9401d33
Add the skills discovery probe kit design (#961)
alexeyzimarev Sep 16, 2026
9110d8c
Add the skills discovery probe kit implementation plan (#961)
alexeyzimarev Sep 16, 2026
ebafc21
Scaffold the skills discovery probe kit (#961)
alexeyzimarev Sep 16, 2026
ebf32bb
Add the probe skill, prompts and reply parser (#961)
alexeyzimarev Sep 16, 2026
32bb8b6
Add the probe sandbox with isolated vendor roots (#961)
alexeyzimarev Sep 16, 2026
6c17b9c
Add the Git exclusion arms to the probe kit (#961)
alexeyzimarev Sep 16, 2026
eb51541
Add verdict rules to the probe kit (#961)
alexeyzimarev Sep 16, 2026
13e1722
Add the run recorder and matrix emission (#961)
alexeyzimarev Sep 16, 2026
f5ed762
Add the startup hook payload to the probe kit (#961)
alexeyzimarev Sep 16, 2026
d68e2de
Add the adapter contract, fake adapter and print driver (#961)
alexeyzimarev Sep 16, 2026
9a1293a
Add the probe orchestrator with scenarios S0 to S4 (#961)
alexeyzimarev Sep 16, 2026
59eb2f6
Harden the probe orchestrator's S2 and S4 bookkeeping (#961)
alexeyzimarev Sep 16, 2026
cc9717d
Add the ACP one-turn driver and fake vendor servers (#961)
alexeyzimarev Sep 16, 2026
4d4cc13
Add the Codex app-server and Pi RPC drivers (#961)
alexeyzimarev Sep 16, 2026
c03bd7e
Close driver handles and test the hook-trust seeding path (#961)
alexeyzimarev Sep 16, 2026
caae66c
Contain arm failures and resume finished probe arms (#961)
alexeyzimarev Sep 16, 2026
b02eed7
Keep per-version matrix rows and the sandbox stderr logs (#961)
alexeyzimarev Sep 16, 2026
092db30
Record one S4 row per root and confirm a miss once (#961)
alexeyzimarev Sep 16, 2026
aa6fa5c
Count tool use and sharpen the drivers' timing evidence (#961)
alexeyzimarev Sep 16, 2026
87cf4c5
Insulate sandbox git and firm up the adapter contract (#961)
alexeyzimarev Sep 16, 2026
5bcef7b
Keep blocked and CLI-level rows idempotent across sweeps (#961)
alexeyzimarev Sep 16, 2026
71749c1
Re-measure untested arms on resume and keep a failing arm's stderr (#…
alexeyzimarev Sep 16, 2026
c1a40ec
Add the Claude probe adapter and judge by the expected skill name (#961)
alexeyzimarev Sep 16, 2026
1f2caec
Add the Codex probe adapter and keep each run's raw event stream (#961)
alexeyzimarev Sep 16, 2026
89741f3
Add the Gemini probe adapter and record a failed vendor run as untest…
alexeyzimarev Sep 16, 2026
dcb54ea
Add the Pi probe adapter and kill a vendor's whole session on shutdow…
alexeyzimarev Sep 16, 2026
35fb4f7
Add the Copilot probe adapter (#961)
alexeyzimarev Sep 16, 2026
f496e37
Add the Kiro and OpenCode probe adapters (#961)
alexeyzimarev Sep 16, 2026
91d3880
Add the Cursor and Antigravity CLI probe adapters (#961)
alexeyzimarev Sep 16, 2026
4c0032d
Render the capability matrix from the probe results (#961)
alexeyzimarev Sep 16, 2026
2618e43
Record pass 1 skills discovery findings and capability matrix (#961)
alexeyzimarev Sep 16, 2026
2c2954d
Run Cursor and agy probes on the real home with project-level hooks (…
alexeyzimarev Sep 16, 2026
746b447
Classify Cursor's ACP tool calls as listed reads or searches (#961)
alexeyzimarev Sep 16, 2026
e9d423b
Give a gated probe arm the reason that applies now (#961)
alexeyzimarev Sep 16, 2026
81c54f0
Probe Cursor's user-level hooks file for one turn and restore it (#961)
alexeyzimarev Sep 16, 2026
ff773b9
Read Cursor's print-mode answer from its event stream (#961)
alexeyzimarev Sep 16, 2026
6400d4a
Classify Kiro's ACP tool calls as listed reads or searches (#961)
alexeyzimarev Sep 16, 2026
3fb7dc7
Drive OpenCode V2 through its own flags and stop its service after a …
alexeyzimarev Sep 16, 2026
016c781
Clone Kiro's default agent for the probe agents so they keep their to…
alexeyzimarev Sep 16, 2026
1c53ff6
Extend the skills discovery findings to all nine harnesses (#961)
alexeyzimarev Sep 16, 2026
e0524a2
Record the full pass 1 matrix across all nine harnesses (#961)
alexeyzimarev Sep 16, 2026
8e40b6c
Probe skill lifecycle, resume and interactive mode across harnesses (…
alexeyzimarev Sep 17, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions docs/probes/2026-09-16-skills-discovery/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
out/
*.stderr.log
*-tui.log
40 changes: 40 additions & 0 deletions docs/probes/2026-09-16-skills-discovery/capability-matrix.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# Capability matrix β€” repo-local skill discovery

Generated by `report.py` from `matrix.json`; do not edit by hand. A `β€”` cell means the entry
could not run (see its status line below and `findings.md`).

| Entry | Version tested | Modes | Native root | Roots consumed | Startup mechanism proven | Exclusion preserving load | Vendor-isolated destination | Reload path | Live catalogue | Startup rewrite | Resume | Nested cwd | Worktree | Peer hook | Interactive | GUI status | Minimum version |
| -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- |
| agy | 1.2.4, 1.2.5 | print | .agents/skills | β€” | β€” | β€” | β€” | β€” | β€” | β€” | β€” | β€” | β€” | β€” | β€” | Antigravity IDE: untested (manual procedure in findings.md) | β€” |
| agy-clidir | 1.2.4 | print | .agents/skills | .agent/skills, .agents/skills | n/a (not run) | gitignore, info-exclude | .agent/skills | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | Antigravity IDE: untested (manual procedure in findings.md) | 1.2.4 |
| agy-dirlayout | 1.2.4, 1.2.5 | print, tui | .agents/skills | .agent/skills, .agents/skills | none | gitignore, info-exclude | .agent/skills | none | n/a (not run) | delete=stale; update=visible_first_turn | add=visible_first_turn; update=visible_first_turn | ancestor=visible_first_turn; local=visible_first_turn | linked-other=not_visible; linked-own=visible_first_turn | n/a (not run) | S1=visible_first_turn; hook-adds-skill=not_visible; add=visible_live | Antigravity IDE: untested (manual procedure in findings.md) | 1.2.4 |
| claude | 2.1.273 (Claude Code), 2.1.274 (Claude Code) | print, tui | .claude/skills | .claude/skills | --settings file over the real config root hooks.SessionStart (tui) | gitignore, info-exclude | none | none | n/a (not run) | delete=revoked; update=stale | add=visible_first_turn | ancestor=visible_first_turn; local=visible_first_turn | linked-other=not_visible; linked-own=visible_first_turn | n/a (not run) | S1=visible_first_turn; hook-adds-skill=visible_first_turn; add=not_visible | n/a | 2.1.273 (Claude Code) |
| codex | codex-cli 0.154.0 | daemon, print, tui | .agents/skills | .agents/skills, .codex/skills | none | gitignore, info-exclude | none | none | add=visible_live; delete=revoked; update=visible_live | delete=not_visible/stale; update=visible_first_turn | add=visible_first_turn; update=visible_first_turn | ancestor=visible_first_turn; local=visible_first_turn | linked-other=not_visible; linked-own=visible_first_turn | hook-from-peer=visible_live | S1=visible_first_turn; hook-adds-skill=not_visible; add=visible_live | n/a | codex-cli 0.154.0 |
| copilot | GitHub Copilot CLI 1.0.85. | daemon, print, tui | .github/skills | .agents/skills, .claude/skills, .github/skills | none | gitignore, info-exclude | .github/skills | /skills reload | add=not_visible; delete=stale; update=stale | delete=revoked; update=visible_first_turn | add=visible_first_turn; update=stale | ancestor=visible_first_turn; local=visible_first_turn | linked-other=not_visible; linked-own=visible_first_turn | hook-from-peer=not_visible | S1=visible_first_turn; hook-adds-skill=not_visible; add=not_visible; reload=visible_after_reload | n/a | GitHub Copilot CLI 1.0.85. |
| cursor | 2026.09.15-d2fe57e | daemon, print, tui | .cursor/skills | .agents/skills, .claude/skills, .codex/skills, .cursor/skills | project .cursor/hooks.json workspaceOpen pluginPaths (print) | gitignore, info-exclude | .cursor/skills | project .cursor/hooks.json workspaceOpen pluginPaths | add=not_visible; delete=revoked; update=visible_live | delete=revoked; update=visible_first_turn | add=not_visible; update=visible_first_turn | ancestor=not_visible; local=visible_first_turn | linked-other=not_visible; linked-own=visible_first_turn | hook-from-peer=not_visible | S1=visible_first_turn; hook-adds-skill=not_visible | Cursor desktop: untested (manual procedure in findings.md) | 2026.09.15-d2fe57e |
| cursor-userhooks | 2026.09.15-d2fe57e | daemon, print | .cursor/skills | n/a (not run) | user ~/.cursor/hooks.json workspaceOpen (print) | n/a (not run) | n/a (not run) | user ~/.cursor/hooks.json workspaceOpen | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a | 2026.09.15-d2fe57e |
| gemini | 0.60.0 | daemon, print, tui | .gemini/skills | β€” | β€” | β€” | β€” | β€” | β€” | β€” | β€” | β€” | β€” | β€” | β€” | n/a | β€” |
| kiro | kiro-cli 2.21.4 | daemon, print, tui | .kiro/skills | .kiro/skills | agent hooks.agentSpawn (cli 2.x) (daemon); agent hooks.agentSpawn (cli 2.x) (print); agent hooks.agentSpawn (cli 2.x) (tui) | gitignore, info-exclude | .kiro/skills | none | add=visible_live; delete=revoked; update=visible_live | delete=revoked; update=visible_first_turn | add=visible_first_turn; update=visible_first_turn | ancestor=not_visible; local=visible_first_turn | linked-other=not_visible; linked-own=visible_first_turn | hook-from-peer=visible_live | S1=visible_first_turn; hook-adds-skill=visible_first_turn; add=visible_live | Kiro IDE: untested (manual procedure in findings.md) | kiro-cli 2.21.4 |
| kiro-agent-bare | kiro-cli 2.21.4 | daemon, print | .kiro/skills | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a | kiro-cli 2.21.4 |
| kiro-agent-skills | kiro-cli 2.21.4 | daemon, print | .kiro/skills | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a (not run) | n/a | kiro-cli 2.21.4 |
| opencode-v1 | 1.18.31 | daemon, print, tui | .opencode/skills | .agents/skills, .claude/skills, .opencode/skills | plugin event session.created (print) | gitignore, info-exclude | .opencode/skills | none | add=not_visible; delete=stale; update=stale | delete=revoked; update=stale/visible_first_turn | add=visible_first_turn; update=visible_first_turn | ancestor=visible_first_turn; local=visible_first_turn | linked-other=not_visible; linked-own=visible_first_turn | hook-from-peer=not_visible | S1=visible_first_turn; add=not_visible | n/a | 1.18.31 |
| opencode-v2 | opencode v2.0.4 | daemon, print, tui | .opencode/skills | .agents/skills, .claude/skills, .opencode/skills | n/a (not run) | gitignore, info-exclude | .opencode/skills | n/a (not run) | add=not_visible; delete=revoked; update=stale | n/a (not run) | add=visible_first_turn; update=visible_first_turn | ancestor=visible_first_turn; local=visible_first_turn | linked-other=not_visible; linked-own=visible_first_turn | n/a (not run) | S1=visible_first_turn; add=visible_live | n/a | opencode v2.0.4 |
| pi | 0.85.1 | daemon, print, tui | .pi/skills | .agents/skills, .pi/skills | extension session_start + resources_discover skillPaths (daemon); extension session_start + resources_discover skillPaths (print) | gitignore, info-exclude | .pi/skills | extension session_start + resources_discover skillPaths | add=not_visible; delete=not_visible; update=visible_live | delete=revoked; update=visible_first_turn | add=visible_first_turn; update=visible_first_turn | ancestor=not_visible; local=visible_first_turn | linked-other=not_visible; linked-own=visible_first_turn | hook-from-peer=not_visible | S1=visible_first_turn; hook-adds-skill=not_visible; add=not_visible | n/a | 0.85.1 |

## Status per entry

- `agy`: S1 failed: the native root's skill was not loaded
- `agy-clidir`: measured
- `agy-dirlayout`: measured
- `claude`: measured
- `codex`: measured
- `copilot`: measured
- `cursor`: measured
- `cursor-userhooks`: measured
- `gemini`: untested: exception=RuntimeError("vendor exit 1 with no reply; extract failed: JSONDecodeE; exception=RuntimeError('no reply read from the screen; exit None; timeout no PRO; exception=RuntimeError('vendor exit -9 with no reply; session/new failed: {"json
- `kiro`: measured
- `kiro-agent-bare`: measured
- `kiro-agent-skills`: measured
- `opencode-v1`: measured
- `opencode-v2`: measured
- `pi`: measured
Loading