Repository navigation
Coding agents: hooks for Kiro CLI (#798) - #825
Conversation
a03a81b to
22b3458
Compare
|
The done-when session, with Kiro CLI 3.3.0 on Windows 11 and Tracekit at this PR's head (22b3458, rebased onto main): The prompt asked for three calls, one each, with no retry: write
Then
The 12 records in the bundle, in order: The resumed prompt stayed in the same run, because Kiro keeps the session id on Before this I'd run the same session at the pre-rebase head (a03a81b), with the same outcome. I rebased because main had gained the gptme, OpenHands, Copilot CLI and CodeBuddy hooks. The conflicts were all in shared lists (harness tuples, docs tables, the TK-F009 and TK-F015 dot-folder groups), and Kiro now sits beside the new entries. Main already had Not covered: macOS and Linux (no Kiro sign-in there), and |
|
Reviewed (read against main, nothing run). The recorded V3 file, shell, fetch and MCP calls map right, and a malformed payload or an unreachable signer blocks with exit 2. Before this leaves draft:
The README counts and the coding-agents column are right. CI is green apart from the Windows test that fails on main too (#827). |
|
Thanks @LoopGlitch26, that's a thorough review. I've moved this back to draft and will work through all seven points. On point 6: PostToolUse, UserPromptSubmit, SessionStart and Stop were recorded in the same session, but I built them in code in the test instead of taking them from the recording. I'll put the recorded ones into |
- Tools: remote_web_search is a WebSearch and invoke_sub_agent a Task (a sub-agent's own calls fire the session's hooks, recorded). The code tool's recorded reads are a Read of their file and pattern_rewrite an Edit; any other operation, a rewrite naming no file, createHook, subagent/<agent> and kiro_powers are denied in the hook. The 2.x names are dropped: they are no hook ids on CLI V3. - Windows: the v2 PreToolUse command ends in a cmd.exe fallback that turns any status but 0 and 2 into 2, as install._blocking does elsewhere. - init refuses a Kiro CLI before 3.0 on both paths; Tracekit's hooks are told by their command, not their name; v1 commands name their event; a listing or search without a path isn't denied. - doctor: the version check on both paths, and a Tracekit hook set to "enabled": false. - Tests take the recorded SessionStart, UserPromptSubmit, PostToolUse and Stop payloads (kiro_events.json) and the newly recorded calls; built ones are labelled as built. - Docs: limits (PowerShell isn't read, so .kiro isn't protected from execute_pwsh), doctor.md, the coding-agents table, and the setup skill (tracekit-skills#3, pinned). Signed-off-by: Khushi Jain <jainkhushibts@gmail.com>
22b3458 to
e7c001f
Compare
|
@LoopGlitch26 all seven points are addressed in e7c001f, rebased onto cda2997. I recorded two more short Kiro 3.3.0 sessions for them.
Live check (Kiro CLI 3.3.0, Windows 11, Tracekit at e7c001f,
CI on my fork is in the description. |
|
Re-reviewed e7c001f: all seven points are fixed. The unrecorded tools are denied, a Windows hook that can't start blocks, pre-3.0 is refused on both paths, doctor sees
Also, |
tracekit init --dev [--v2] --agent kiro writes Tracekit's own hook file, ~/.kiro/hooks/tracekit.json (--project: the project's .kiro/hooks), one hook per event with a 600 s PreToolUse timeout; uninstall removes it, and a file there with a hook that isn't Tracekit's is left alone. init and doctor read kiro-cli --version and say when a CLI before 3.0 won't read ~/.kiro/hooks. The mapping follows payloads recorded from Kiro CLI 3.3.0 (tests/data/harness/kiro.json): Claude Code's payload shape with no call id (paired by tool and arguments), the V3 tool ids, each file tool decided on the key it reads (delete_file's targetFile), and mcp_<server>_<tool> kept whole, since the split isn't known. A file, shell or fetch call without the argument it is decided on is denied in the hook. The coding pack holds an agent's edit of .kiro/ (TK-F009, TK-F015; 2026.11-v2-82). Docs: a Kiro CLI section and column in coding-agents.md, and limits.md rows for what the recording settled: Kiro runs a call whose hook exits other than 2, can't start or times out, and fires no SessionEnd in --no-interactive. Signed-off-by: Khushi Jain <jainkhushibts@gmail.com>
…lone; tests and eval hashes The generic installer tests treat Kiro's file as Tracekit's own, as they do goose's and Cline's, and the fail-closed test now runs Kiro's commands too. Evals: the coding pack's policy hash at 2026.11-v2-82 (no result changed). Signed-off-by: Khushi Jain <jainkhushibts@gmail.com>
- Tools: remote_web_search is a WebSearch and invoke_sub_agent a Task (a sub-agent's own calls fire the session's hooks, recorded). The code tool's recorded reads are a Read of their file and pattern_rewrite an Edit; any other operation, a rewrite naming no file, createHook, subagent/<agent> and kiro_powers are denied in the hook. The 2.x names are dropped: they are no hook ids on CLI V3. - Windows: the v2 PreToolUse command ends in a cmd.exe fallback that turns any status but 0 and 2 into 2, as install._blocking does elsewhere. - init refuses a Kiro CLI before 3.0 on both paths; Tracekit's hooks are told by their command, not their name; v1 commands name their event; a listing or search without a path isn't denied. - doctor: the version check on both paths, and a Tracekit hook set to "enabled": false. - Tests take the recorded SessionStart, UserPromptSubmit, PostToolUse and Stop payloads (kiro_events.json) and the newly recorded calls; built ones are labelled as built. - Docs: limits (PowerShell isn't read, so .kiro isn't protected from execute_pwsh), doctor.md, the coding-agents table, and the setup skill (tracekit-skills#3, pinned). Signed-off-by: Khushi Jain <jainkhushibts@gmail.com>
e7c001f to
bf3cb8a
Compare
…t of the folder (Cygnux-Labs#825) invoke_sub_agent hands the sub-agent its contextFiles ([{"path": ...}]) with no read_file of them (recorded on 3.3.0), so a file there was read undecided. A call that names any is denied in the hook: the agent names the files in the prompt, and the sub-agent's read of each is decided. code's pattern_rewrite takes a folder (`path`), decided as an Edit of it, so a folder inside .kiro is held by TK-F009. On 3.3.0 it only previews ("Always previews"); recorded on a folder, it changed no file. limits.md: `path` doesn't keep the search to that folder. Signed-off-by: Khushi Jain <jainkhushibts@gmail.com>
…t of the folder (Cygnux-Labs#825) invoke_sub_agent hands the sub-agent its contextFiles ([{"path": ...}]) with no read_file of them (recorded on 3.3.0), so a file there was read undecided. A call that names any is denied in the hook: the agent names the files in the prompt, and the sub-agent's read of each is decided. code's pattern_rewrite takes a folder (`path`), decided as an Edit of it, so a folder inside .kiro is held by TK-F009. On 3.3.0 it only previews ("Always previews"); recorded on a folder, it changed no file. limits.md: `path` doesn't keep the search to that folder. Signed-off-by: Khushi Jain <jainkhushibts@gmail.com>
bf3cb8a to
31982ed
Compare
|
Re-reviewed at 31982ed. Both points from the last review are settled, and this is ready from our side.
I also checked the Kiro tools that could start a process. No V3 CLI tool adds or connects an MCP server by command. The files that do start one are all under Two things for later, not for this PR. Next steps:
|
Rule ids: #848 and #825 add none (they extend TK-F009's and the effects rule's harness folders), so the train's new ids stay TK-A079 and TK-A080 (#789) and TK-A081 (#844's RemoteTrigger); nothing collides. Packs: still one step above main's, each once: coding 2026.11-v2-83, dev 2026.11-dev-6, server 2026.11-server-18, strict 2026.11-strict-12. Harness lists: AGENTS.md names Kiro CLI beside jcode (#825 didn't). Evals re-recorded once more, by their own scripts, run on the host with main's own code (4ce4aa7) and the train's packs, corpora and results files as data: - E17: 951 of 951 harmful calls stopped, 1 false alarm in 971. - E21 policy robustness: held-out 286 / 420 default and 312 / 420 strict, as before; strict's E17 false positives 50 of 971 (5.1%); the recorded gates are main's. - E21 robustness: all 951 originals and all 2,948 rewrites contained. The latency rows in docs/evaluation.md are this run's, on macOS arm64 (Apple M4 Pro), Python 3.11.15, the machine the gates were recorded on; every p99_ref is within its cap. README, docs/evaluation.md and docs/limits.md quote these numbers. Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
|
Thank you, @LoopGlitch26, and thanks for checking the Kiro tools that could start a process. I've noted |
Fixes #798.
tracekit init --dev [--v2] --agent kirohooks Kiro CLI into Tracekit, with uninstall and doctor support. The mapping follows payloads recorded from a real Kiro CLI 3.3.0 session on Windows 11 (tests/data/harness/kiro.json).What the recording settled (the issue's questions)
kiro-cli 3.3.0(KIRO_CUSTOM_USER_AGENT:KiroCLI/3.3.0 … os/windows).execute_pwshon Windows,fs_write,str_replace, …). The 2.x names (shell,read,write) aren't: Kiro's V3 catalog lists them as 2.x names, not hook ids, and init refuses a 2.x CLI.tool_input:execute_pwsh{command, description, cwd, run_in_background, timeout}fs_writeandfs_append{path, text}str_replace{path, oldStr, newStr, replace_all}read_file{path, offset, limit}list_directory{path, explanation, depth}grep_search{query, caseSensitive, excludePattern, explanation, includePattern, path}file_search{explanation, query, excludePattern, includeIgnoredFiles}web_fetch{url, mode, searchPhrase}delete_file{explanation, targetFile}: notpathPreToolUseorPostToolUse. Sokirois inPAIRED, and calls are paired by tool and arguments, as for Gemini CLI, Droid and Devin.rec_serverwith a toolecho_textreaches the hook asmcp_rec_server_echo_text. The split really is ambiguous, so the call is decided asmcp__rec_server_echo_text, whole, as Gemini CLI's and Crush's are. Before a deferred MCP tool's first call, Kiro callstool_loadwith{"tool_ids": ["rec_server::echo_text"]}. That id is unambiguous, but it isn't the call itself.remote_web_search{query};invoke_sub_agent{name, prompt, explanation}, whose sub-agent's own calls fire the session's hooks;code{operation, …}with its file inpath(lookup_symbols) orfile_path(get_document_symbols,pattern_search,find_references,diagnostics, andpattern_rewrite{pattern, replacement});createHook{actionType, command, id, name, trigger}; andtool_load{"tool_ids": ["builtin::<tool>"]}before a deferred built-in.SessionStart,UserPromptSubmit(prompt),PreToolUse,PostToolUse(tool_response, a string) andStop.SessionEnddidn't fire inkiro-cli chat --no-interactive. Kiro runs hook commands under cmd.exe on Windows.PostToolUsefor the blocked call. Exit 1 doesn't block, and a hook command that fails to start doesn't block either.timeout: 0waits as long as the hook takes.Change
agent_hooks.py:kiroinAGENT_NAMES(recorded askiro-cli).KIRO_TOOLSandKIRO_PATH_ARG: each file tool is decided on the key it reads (delete_file'stargetFile), never a decoyfile_path,pathorpathsbeside it.fs_write/fs_appendtextis a Write'scontent;str_replaceoldStr/newStrareold_string/new_string.normalise()takes Claude Code's payload path, paired._blockeddenies a file, shell or fetch call without the argument it's decided on (KIRO_DECIDED_ON), before the signer, as Devin's are; a listing or a search without a path reads the working directory.remote_web_searchis aWebSearch,invoke_sub_agentaTask.code's recorded reads are aReadof their file andpattern_rewriteanEditof it (bothfile_pathandpathdecided)._blocked_kirodeniescreateHook,subagent/<agent>,kiro_powers, anycodeoperation not recorded, and a rewrite that names no file.~/.kiro/hooks/tracekit.json(--project:.kiro/hooks/tracekit.json).tracekit-<event>, with no matcher (every tool, as recorded).PreToolUsetimeout 600 s, the rest 30 s. I chose 600 over0: Tracekit's hook answers within 540 s, and0would let a hung hook stop Kiro forever.SettingsError).install._blockingalready turns a hook that can't run into exit 2. On Windows, Kiro runs commands under cmd.exe, so_commandkeeps cmd.exe's quoted form, and the v2PreToolUsecommand ends in a cmd.exe fallback that turns any status but 0 and 2 into 2, asinstall._blockingdoes.kiro_version()readskiro-cli --version.initrefuses a CLI before 3.0, which reads no.kiro/hooksfile, user-level or--project;doctor(D-HOOKS-EVENT) warns about one on both paths.doctor:_harness_hooksreads Kiro's list of hooks, less those set to"enabled": false(reported), so the event and timeout checks apply to it..kiroinTK-F009andTK-F015(hooks, agents, MCP servers and settings),2026.11-v2-83(main is at-82). The policy hashes ineval/resultsare updated; no decision changed.coding-agents.md;limits.mdrows for what the recording settled (exit status, timeout, no call id, MCP names, noSessionEnd), and that the policy doesn't read PowerShell, so.kiroisn't protected fromexecute_pwsh;D-HOOKS-EVENTandD-HOOKS-TIMEOUTindoctor.md, and in the setup skill (tracekit-setup: the kiro agent (Kiro CLI 3.0 or later) tracekit-skills#3, whose blobtest_docs.pypins);integrations.md,support.mdand the README lists.Tests
test_harness_hooks_v2.py:test_kiroruns the 22 recorded cases (kiro.json: allow, held and approved, deny,tool_load, MCP,remote_web_search,invoke_sub_agentand its sub-agent's call, thecodeoperations);SessionStart,UserPromptSubmit,PostToolUseandStoppayloads (kiro_events.json), withSessionEndbuilt;createHook, and builtsubagent/…,kiro_powersand unrecordedcodecalls, denied before the signer;PreToolUsehook whose Python is gone exiting 2 under cmd.exe.test_agent_hooks.py:test_doctor_v2.py: a hook set to"enabled": falseand a 2.x CLI are warned about.test_signer_policy.py:.kirois harness config in every pack and engine.test_a_hook_that_cannot_start_blocks_only_a_tool_callruns Kiro's commands too.Rebased onto main (cda2997). On my fork, at this head (e7c001f):
test_cursor_edit_tools_are_decided_as_edits, which fails on main too and is fixed by #838 (#827)eval/resultsLive session (the done-when)
Kiro CLI 3.3.0 on Windows 11, Tracekit at e7c001f,
tracekit init --dev --v2 --agent kiro --project, andkiro-cli chat --no-interactive --trust-all-tools, so Tracekit's hook was the only gate:fs_write notes.txt: allowed, written.execute_pwsh sudo echo hi: denied (TK-D001). Kiro didn't run it, and the hook's stderr held only the reason.fs_write .kiro/hooks/probe.json: held (TK-F009). Aftertracekit approvals approve, a resumed prompt (--resume) retried it, the approval was consumed, and the file was written.createHook: denied in the hook. In-hook denials don't reach the signer, so it isn't in the run.tracekit verifyon the exported run: 4 tool calls in 12 signed records, none changed after signing (the run stays open, since--no-interactivesends noSessionEnd).PreToolUsehook's Python moved away, Kiro blockedecho fail-closed-probe, showing "tracekit: the hook could not run; call blocked (fail-closed)".Not covered: macOS and Linux (no Kiro sign-in there), and
SessionEndin an interactive session.