Repository navigation
Hooks and CLI: fixes from the 1.6.0 pre-release review - #673
Merged
Merged
Conversation
- Devin: exec is decided on its command under any spelling (Command), a tool name in another case (EXEC) is read as that tool, and an exec that sends its command under two spellings is denied in the hook. - Kilo: background_process gets no model receipt, as Codex write_stdin and Qwen monitor don't; docs/limits.md names it and Devin write_to_process. - findings_box: a line nested too deep to parse counts as unreadable instead of crashing analyze --v2. - tips and feedback --harmless: name only the rules that placed a hold, not a flag that also matched (as report._placed places them). - audit: decides each call with the signer's own code (signer.service.evaluate, moved out of SignerService._evaluate), so MCP, bare-name and PowerShell checks count. - feedback --harmless: starts no signer; says when none is running. - report and audit windows under an hour print minutes or seconds, not 0 hours. - tips: no "calls waiting" pointer when every hold was answered. - doctor: for a settings.json that isn't a JSON object, says to fix it first. - demo: points to the dev-mode line (system mode or Tracekit Box), not v1 system mode. Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
…ent joined
write_to_process was decided on all its argument strings joined into one
command, so a quote opened in another argument hid the text it writes
({"a": "echo '", "input": "rm -rf ~ #'"} was decided as echo). Devin
documents none of its fields: it is now decided on its one argument besides
shell_id, whatever its name, and a call with no text, another argument, or a
name sent twice in any case is denied in the hook.
Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
…doctor's advice for an unreadable file
- A Devin call whose tool_input sends an argument name twice (in any case,
with any value) is denied in the hook: json.loads keeps the last value and
a Go-style decoder may keep another ({"command": ..., "command": null} ran).
The raw payload is read with _Pairs, as for Crush and agy; this replaces
exec's own two-spellings check.
- write_to_process types into a live terminal, whose line editor acts on
control characters: text holding a C0 control other than a line end, or
DEL, is denied in the hook; \r is read as a line end.
- doctor: a settings.json it can't read is told to be made readable, not to
become a JSON object.
Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
…s a line end - An exec whose input isn't the documented object (a plain string), or that names no command, was decided on nothing and ran; it is now denied in the hook, as the other Devin inputs with nothing to decide on are. - Devin may type exec's command into a terminal, where \r ends a line: it is read as a line end, so each part is decided. Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
…rs in exec - write_to_process: interactive bash expands history in a typed line before it runs it (after an allowed line, "!-1" or "!:1*" ran it again, and "!#:1-5" ran a push from its own line). Typed text with a ! before anything but a blank, a line end, = or (, or a ^ starting a line, is denied in the hook. C1 controls are denied there too. - exec: a C0 or C1 control (but a tab or a line end) or DEL in its command is denied, as Devin may type it into a terminal. A ! in it stays decided as written (git commit -m 'fix!', wait $!); docs/limits.md says so. Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
In the shell write_to_process types into, one line can set state that changes what a later line runs, while the policy decides one call at a time: after ": git push --force origin main", "fc -s ': =' :" re-ran the push, as "history -s '...'" then "fc -s" did. Typed text is read as the policy's shell parser reads it, and denied in the hook when it replays history (fc; history -s, -r or -n), defines an alias or a function, sets a trap, a key binding, a builtin, a shell option (set, shopt) or a hashed path, sources a file, or sets PROMPT_COMMAND, PS0 to PS4, PATH or BASH_ENV. The parser now also reports the names of the functions a line defines. docs/limits.md names the class, and that exec isn't checked for it. Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
…ow set and alias - write_to_process text that uses the exec builtin is denied: exec 0<file makes the shell read commands from a file (bash 3.2 ran its lines), and exec can replace the shell. - Setting LD_PRELOAD, LD_LIBRARY_PATH, LD_AUDIT or any DYLD_* is denied: every later program would load that code. - set is denied only when it changes options (an argument starting with - or +), so psql's "set search_path to public;" passes; alias only when it defines one (name=...), so a Python REPL's "alias = 3" passes. - docs/limits.md names the general limit: a persistent shell's state set by an earlier line can change what a later line runs; the forms Tracekit knows are denied, the rest isn't tracked, so run Devin in a sandbox or Box where that matters. Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
alias and hash were denied, but not their tables: BASH_ALIASES[ll]=... and BASH_CMDS[ls]=... (also +=, and through export, declare, typeset or readonly). A nameref (declare, typeset or local -n) lets a later assignment set a denied variable under another name, so -n is denied too. Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
LoopGlitch26
added a commit
that referenced
this pull request
Nov 8, 2026
…arser holds it, too #673's check for typed text that sets state a later line runs read the line with the policy's shell parser and left a line it can't read to the signer. This branch's parser holds a state variable set to text it can't name (PS1, BASH_ALIASES, BASH_CMDS), a nameref to one and `hash -p`, so `read PS1` or `BASH_CMDS[ls]=/tmp/x` were held, not denied (tests/test_agent_hooks.py's typed-state test). Such a line is now read as the engine reads an unreadable one, each piece's words as written, with an assignment as its first word; one with no state word still goes to the signer. Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
LoopGlitch26
added a commit
that referenced
this pull request
Nov 8, 2026
…e it decides, and stays fast (#664) * Policy: the coding pack reads more of what a shell command does before it decides, and stays fast * Tests and E21: plain descriptions in the new tests, and one consistent note on the latency caps * Policy: read a process call's arguments a run at a time, and decode only literals with an escape A process call's arguments in inline code were read one character at a time, and every plain literal went through ast.literal_eval, so 64 KiB of subprocess.run([ took 7 to 10 times main's time to decide. Runs where no literal can start are now read in one step, and a literal with no backslash or carriage return is its own text. Verdicts are unchanged. Also: the decide-time child times with perf_counter (Windows' monotonic clock read a fast decision as 0 s), and E21's robustness result records the current policy hash. * Policy: one work budget per decision, shared by everything the decision reads A decision's sub-reads each had a budget of their own: each argument read as a command was a full decision with a fresh budget (6,500 quoted arguments took 0.75 s), a line the parser couldn't read was read again in pieces on a fresh budget, and a function body's dry read had its own. The parse, its other readings, what is read of an unreadable line and each argument read as a command (128 more each) now spend one budget and one fallback deadline. Once the budget is spent nothing more is read and the call is held (TK-SHELL-PARSE, cause budget), never allowed. A head the decision can't afford to read again isn't read again; an arithmetic expansion costs what a command does; an argument with no letter (blank escapes aside) and no $'...' isn't read as a command; the argv form with WORD_SPACE is built only where a word before the last has a space, so sqlite3's SQL is searched once. * Policy: decide long arguments, quoted text and SELECTs faster, with the same verdicts - shell: read a run of plain arguments, a run of blanks, and double-quoted or heredoc text in one step each - classes: read a run of SQL text that opens no comment, literal or quote in one step - compile and engine: a pattern whose top level is a group of alternatives is searched only on a subject that holds a string of each set of one alternative (requirements): TK-A039's sqlite3 branch isn't searched on a SELECT - engine: look for a dot or // before the dotted-path search; don't combine the ways of no expansion - tests: the pipeline that spends the whole work budget is held to 2 s (CI's py3.9 Linux runner took 1.07 s); the prefilter tests check requirements too, and TK-A039 is not searched on a SELECT The corpora (every pack, both engines), the week's 39,890 calls, E17 and the parser oracle decide as before. Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com> * Policy: look for a pattern character before searching each name for a pattern The Tracekit-paths reading searched each of a command's words for a pathname pattern; a line of 21,800 words spent 3 ms on it. A word with no *, ? or [ holds none, so the words joined are checked first. Decisions are unchanged on the corpora (every pack, both engines) and the week's 39,890 calls. Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com> * Policy: check a rule's alternatives' strings only on a subject of 1 KiB or more Checking them on every subject made E21's calls decide 2-4% slower at p99 (CI's Linux runner then measured default/regex at 1,092 us, over its 1 ms check); a short subject's search costs no more than the check. With the check from 1 KiB, E21's p99 is back at the previous commit's on both engines, and a 60 KiB SELECT still isn't searched for TK-A039. Decisions are unchanged on the corpora (every pack, both engines), the week's calls and E17. Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com> * Tests: hold the database-statement speed tests to the decision budget's bound on slow runners Both decisions read the statement as a command, which the decision's budget caps: about 70-80 ms on a laptop, and about 10x that on CI's Windows runners (0.737 s for the mongosh case on py3.13, over the 0.5 s bound). They are held to 2 s, as the budget-spending pipeline is; a runaway search still fails them, as nondeterministic. The other speed tests stay within their bounds at 10x their laptop times, or scale with the machine. Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com> * Tests: scale the budget-capped speed bounds by the machine, as the other speed tests do The mongosh whitespace, indented-SELECT and long-pipeline bounds were a flat 2 s. Each is now about five times its laptop time, scaled by machine_scale (1 to 4): 0.4 s for the statement read as a command (up to 80 ms on a laptop), 0.35 s for the SELECT (about 70 ms) and 1.25 s for the pipeline that spends the whole work budget (about 0.26 s). A runner ten times slower than the laptop passes where it runs the scale's search at least 2.5 times slower. Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com> * Hooks: a typed line that sets shell state is denied when the policy parser holds it, too #673's check for typed text that sets state a later line runs read the line with the policy's shell parser and left a line it can't read to the signer. This branch's parser holds a state variable set to text it can't name (PS1, BASH_ALIASES, BASH_CMDS), a nameref to one and `hash -p`, so `read PS1` or `BASH_CMDS[ls]=/tmp/x` were held, not denied (tests/test_agent_hooks.py's typed-state test). Such a line is now read as the engine reads an unreadable one, each piece's words as written, with an assignment as its first word; one with no state word still goes to the signer. Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com> * Policy: the prefilters read a subject casefolded, as RE2 folds it for a rule matched in any case TK-D007 matches in any case, and RE2 folds a long s (U+017F) to s and the Kelvin sign to k. The prefilters read a subject lower-cased, which keeps both, so a read of ~/.tracekit-ſigner/… past 1 KiB held none of TK-D007's alternatives' strings, wasn't searched, and was allowed where d7beb4c denied it (the review of dde5208). casefold() maps those two to the same letters and keeps every ASCII letter lower() makes, so a subject is read casefolded; a requirement string that isn't ASCII gives its rule no requirements. Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com> * Policy: the fallback engine matches a rule in any case as RE2 does, a long s as s and the Kelvin sign as k RE2's case-insensitive match folds U+017F to s and U+212A to k, as macOS's file system does; the fallback engine's ASCII mode folds neither. TK-D007 still held most such reads through their casefolded path forms, but not one whose casefolded form is the installed package's (`~/.tracekit-ſigner/dist-packages/tracekit/…`: st is st), which RE2 denied and the fallback allowed. A pattern matched in any case is now also searched, on the fallback, with those two characters as their ASCII letters, within the decision's deadline (two linear searches at most). Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com> * Tests: give the database-statement speed tests room for a macOS runner fifteen times the laptop On CI's macOS py3.13 runner the mongosh whitespace case took 1.217 s, about 15 times its laptop time, while machine_scale read 1.6: this work, reading the statement as a command, slows more there than the regex search the scale times. Both database-statement tests now allow 1.0 s times the scale, about twelve times the laptop at a scale of 1, so a runner 15 to 20 times slower passes. The pipeline test keeps its bound. Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com> * CI: gate E21 latency on the runner's own reference time, not absolute microseconds Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com> --------- Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes from the 1.6.0 pre-release review of the coding-agent hooks and CLI messages. Each item has a regression test that failed before the fix.
C1 (high): Devin
execwas decided only on a lower-casecommandkey.{"Command": "git push --force origin main"}ran, and beside a lower-case decoy the call was decided on the decoy.EXECpassed as an unknown tool.Fix:
execis decided on its command under any spelling; a Devin tool name in another case is read as that tool; anexecthat sends its command under two spellings is denied in the hook. docs/limits.md says so.Test:
test_agent_hooks.py::DevinTools::test_exec_is_decided_on_its_command_under_any_spelling(v1 and v2 engines, and_blocked). Also checked through the v2 hook against a real in-process signer.C2 (medium): Kilo's
background_processwas treated as a finished call, so its model receipt hashed the file before the background write.Fix: it is marked
later, as Codexwrite_stdin, Devinwrite_to_processand Qwenmonitorare. The docs/limits.md row names Kilo and Devin.Test:
test_model_receipt.py::HookRecords::test_kilos_background_process_gets_no_receipt.C3 (low): findings_box
_detachescaught onlyParseError. A line nested too deep raisedRecursionErrorup throughanalyze --v2.Fix: it counts as unreadable.
Test:
test_findings_box.py::Background::test_a_line_nested_too_deep_to_read_is_unreadable_not_a_crash.B1, B4:
tracekit tips(and the viewer's TIPS page) and thefeedback --harmlesstitle named a flag rule that only also matched as a rule that held the call.Fix: only the rules that placed the hold are counted, as
report._placedplaces them; feedback lists the others as "also matched, not why".Tests:
test_tips.py::Holds::test_a_rule_that_only_also_matched_is_not_named_as_one_that_held,test_feedback.py::TestHarmless::test_a_rule_that_only_also_matched_is_not_named_as_one_that_held.B2:
auditreplayed withengine.decide(tool, args), missing the signer'smcp__<server>__<tool>mapping, bare-name and PowerShell checks, so the week summary undercounted holds.Fix: the signer's
SignerService._evaluatebody moves tosigner.service.evaluate(policy, ...); the signer and audit both call it.Test:
test_audit.py::TestAudit::test_each_call_is_decided_as_the_signer_decides_it.B3:
feedback --harmless IDwith no signer running started a dev signer (keys and a store), then saidunknown_approval.Fix: it reads from the dev signer only if one is running, and otherwise says no signer is running.
Test:
test_feedback.py::TestHarmless::test_with_no_signer_running_it_starts_none_and_says_so.B5: report and audit windows under about 3 minutes printed "0 hours".
Fix: under an hour they print minutes, and under a minute seconds.
Test:
test_report.py::When::test_a_window_under_an_hour_is_in_minutes_never_0_hours.test_share_is_counts_onlyhad "0 hours" built into its expected output, so it was updated.B6: when every hold was answered, the tips holds card still pointed to waiting calls.
Fix: in that case the card has no fix.
Test:
test_tips.py::Holds::test_when_every_hold_was_answered_it_points_to_no_waiting_call.B7: for a settings.json that isn't a JSON object, doctor suggested
tracekit init --dev --v2, which refuses such a file.Fix: the advice says to make the file a JSON object first.
Test:
test_doctor_v2.py::CodingAgentHooks::test_a_settings_file_that_isnt_an_object_is_fixed_before_init.B8:
tracekit demosuggested v1 system mode, which macOS refuses.Fix: it prints the dev-mode line that
init --devand doctor print.Test:
test_agent_hooks.py::DemoPerAgent::test_it_points_to_a_separate_signer_where_init_and_doctor_do.