Skip to content

Hooks: GitHub Copilot CLI support - #710

Merged
LoopGlitch26 merged 11 commits into
Cygnux-Labs:mainfrom
hossainzarif23:agent/issue-365
Nov 13, 2026
Merged

LoopGlitch26 merged 11 commits into
Cygnux-Labs:mainfrom
hossainzarif23:agent/issue-365

Conversation

@hossainzarif23

Copy link
Copy Markdown
Contributor

Closed #365

What it does

  • Events: preToolUse, postToolUse and postToolUseFailure, paired by Copilot's own toolCallId.
  • Hooks file: Tracekit's own file, ~/.copilot/hooks/tracekit.json (--project: .github/hooks/tracekit.json), with a 600 s timeout on preToolUse. Copilot's default is 30 s.
  • Blocking: exit 2 plus {"permissionDecision": "deny", "permissionDecisionReason": ...} on stdout. Copilot denies on any non-zero exit but shows the reason only from the JSON.
  • Built-in tools: a table pinned to 1.0.107: bash, apply_patch, view, glob, rg, web_fetch, task map to Bash, Edit (every file the patch names, through PATCH_FILE), Read, Glob, Grep, WebFetch, Task. web_search is decided as mcp:github-mcp-server/web_search.
  • MCP tools: the payload doesn't say which tools are MCP ones, and Copilot joins server and tool with a - that either name may contain. Any other name with a - is sent as mcp__<name>. The signer's evaluate already tried each _ split of such a name for Gemini CLI; it now tries each - split as well, plus the bare tool names and the unknown-tool reading, and the strictest decision stands. One signed decision per call.
  • Truncated names: a name of 64 characters or more is denied in the hook, since Copilot cuts names there.
  • preMcpToolCall: not read, as agreed on the issue. It runs after the decision, can't block and isn't documented.

Tests

  • A built-in from the table is decided as its mapped name; a hyphenated name becomes mcp__<name>; a hyphenless unknown name stays itself.
  • A hyphenated server and tool where one split is denied and the others allowed: the call is denied, and the reason is in the JSON reply. This test fails without the signer change.
  • A sanitized dotted tool name (tracekit-space-get-secret) is still held by TK-A037.
  • A 63-character name is decided; 64 and 65 are denied before the signer.
  • A 1.0.105-shaped payload with no toolCallId is blocked.
  • A captured postToolUseFailure payload becomes a failed result.
  • init writes the three events with a 600 s pre-tool timeout, and uninstall keeps a person's own hooks.
  • Copilot is covered by the existing all-harness installer and doctor tests.

Run in a Linux container: the hook, signer-policy, docs, doctor, install and CLI test files passed (590 passed, 10 skipped) before merging main; after the merge the hook, signer-policy and docs files passed (394 passed, 2 skipped). make lint passes. I have not run the full make test, make build or make test-ts.

Checked in a real run

Copilot CLI 1.0.107, Linux, hooks written by tracekit init --dev --v2 --agent copilot --project:

  • a bash call and an MCP call ran and were recorded;
  • an MCP get_secret call was held (TK-A037): Copilot showed "Denied by preToolUse hook: [tracekit] HELD, nothing ran …" and the MCP server never received the call;
  • tracekit doctor reported the hooks present, each naming its event, with a 600 s pre-tool timeout.

That run was before main was merged in.

Limits (added to docs/limits.md)

  • The decision reads Copilot's sanitized, joined name under every split, so a rule for one server can hold another's call, and a rule naming a server or tool with a space or dot doesn't match. The record holds the joined name, not the true server and tool. A name that also contains __ is split there alone.
  • A Copilot tool outside the pinned table is decided as an unknown tool until the table names it.
  • Tool events only: no prompts, session events or model text.
  • Not tested on Windows, in the VS Code agent or in the cloud agent.

Notes for review

  • The hook entries use the command and timeout keys rather than the documented bash and timeoutSec. Copilot 1.0.107 honoured both in a live run (a 40 s hook was not cut at the 30 s default), and it lets the existing installer and doctor code read Copilot's file unchanged. Say if you'd prefer the documented keys.
  • Every custom MCP tool I captured arrived as <server>-<tool>; none arrived under a bare name. So table names are not also decided as MCP names.
  • The hyphen splits also apply to Gemini CLI's and Crush's mcp__ names. Each extra reading can only make a decision stricter.

Signed-off-by: Zarif Hossain <hossainzarif2002@gmail.com>
Signed-off-by: Zarif Hossain <hossainzarif2002@gmail.com>
Signed-off-by: Zarif Hossain <hossainzarif2002@gmail.com>

@LoopGlitch26 LoopGlitch26 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this is careful work: on the v2 hook an unreadable payload, a missing toolCallId, an odd toolName and a non-string apply_patch are all blocked, the deny is exit 2 plus the JSON reason, names of 64+ characters are denied with a clear message (tested at 63/64/65), and uninstall keeps a person's own hooks. The DCO sign-offs are in now too. The signer change reads right: the - splits add no holds on the shipped packs for Gemini or Crush names, and evaluate() is the right place, since tracekit audit replays it. What's left, most important first:

  1. The v1 hook lets every call through. _command (agent_hooks.py:1199) names the event only for v2 and Antigravity, and Copilot's payload doesn't name it, so normalise (:601) returns None and run exits 0. tracekit init --dev --agent copilot (offered at docs/coding-agents.md:18), then any bash call, runs undecided. The v1 tests pass preToolUse on argv, which the installed command doesn't. Name the event for copilot as for Antigravity (and test the installed command line), or refuse v1 for copilot.
  2. Built-in tools missing from the table (:101). Copilot's hooks and CLI references also list create, edit, str_replace_editor, grep, powershell, write_bash/write_powershell and write_agent; with no - in them, each is decided as an unknown tool (flagged, and it runs). For example edit {"path":"~/.copilot/hooks/tracekit.json"} and create {"path":".git/hooks/pre-commit"} meet no file rule, and bash {"command":"bash","mode":"async"} then write_bash {"input":"…"} reaches the shell unread. Your captures (apply_patch, rg) look like the GPT tool set; with a Claude model Copilot uses create/edit/grep, so please capture those too. Map create→Write, edit/str_replace_editor→Edit, grep→Grep, powershell→PowerShell, and write_*→Bash with command = the input, as Codex's write_stdin is.
  3. Each tool's own path argument. rg/glob with paths as a string (the captured shape, test_agent_hooks.py:48) is decided on no path, since the engine reads paths only as a list (engine.py:458): rg {"pattern":"x","paths":"~/.gnupg/private-keys-v1.d/a.key"} gets no TK-A020 hold. And view with a decoy file_path beside path is decided on the decoy. Pin each tool's own argument, as the Crush, goose and Antigravity adapters do.
  4. COPILOT_HOME. With it set, Copilot reads $COPILOT_HOME/hooks/, so Tracekit's hook never runs. Handle it in config_path (:952) and doctor's USER_ENV, as CODEX_HOME is.
  5. Copilot's own config isn't protected. Add .copilot/, .github/hooks/ and .github/copilot/settings*.json to TK-F009/TK-F015 and to the limits row at 174, like every other harness's (disableAllHooks: true in .github/copilot/settings.json turns the hook off). Two limits rows: Copilot also runs a repo's .claude/settings*.json hooks, so a project-level Tracekit Claude Code hook would decide each call twice and block it (as with Devin); and another hook's modifiedArgs can change a call after Tracekit decided it (Crush has the same row, 423).
  6. --project writes .github/hooks/tracekit.json with this machine's Python path. .github/ is usually committed, and teammates and Copilot's cloud agent would read it: the hook can't start there, so every call is denied, and the local path is published. Warn at init and say how to keep it out of the commit.
  7. Smaller:
    • :614 p["toolResult"]["textResultForLlm"] raises KeyError when there's no result; use .get.
    • Copilot's docs say hook timeouts fail open: say in the docs that the hook answers within 540 s, under the 600 s it sets.
    • Doctor assumes a 60 s default timeout; Copilot's is 30.
    • Copilot isn't yet in doctor.md, integrations.md, the harness_hooks docstring or the support table (with the minimum version).
    • Uninstall leaves {"version":1} in Tracekit's own file, and creates that file when nothing was installed.
    • The new tests are module-level pytest functions; AGENTS.md asks for unittest classes.
    • Missing tests: toolArgs sent as a JSON string, a non-string toolName, apply_patch with object args, a missing toolResult.

Items 1 and 2 are the ones that let a call run undecided, so they come first. Happy to look again as soon as you push.

Signed-off-by: Zarif Hossain <hossainzarif2002@gmail.com>

@LoopGlitch26 LoopGlitch26 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this covers all seven: the v1 command names its event (and the test runs the installed command line), the tools Copilot gives other models are mapped, each file tool is decided on its own path or paths, COPILOT_HOME is followed in init and doctor, Copilot's config is flagged in TK-F009 and TK-F015 with the limits rows, --project warns, and the smaller items are in, with the tests in a unittest class.

One thing left: the coding pack's rules changed, so its version has to move with them. Your branch has 2026.10-v2-77, and main is at -78 since #690. Please merge main and set version: '2026.10-v2-79'. If another pack change lands first, I'll move the number on your branch so you don't have to.

I'll look again once that's in and CI is green.

Signed-off-by: Zarif Hossain <hossainzarif2002@gmail.com>
Signed-off-by: Zarif Hossain <hossainzarif2002@gmail.com>
Signed-off-by: Zarif Hossain <hossainzarif2002@gmail.com>
Comment thread README.md Outdated
sessions on this machine; nothing is installed and nothing leaves it.
2. **With their yes, wire your hooks.** `pip install tracekit-ai`, then `tracekit init --dev --v2`, plus
`--agent codex`, `cursor`, `gemini` or another of the 12. In Claude Code, the plugin can take init's place (one or
`--agent codex`, `cursor`, `gemini` or another of the 13. In Claude Code, the plugin can take init's place (one or

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line continues list item 2, so it keeps its three-space indent:

Suggested change
`--agent codex`, `cursor`, `gemini` or another of the 13. In Claude Code, the plugin can take init's place (one or
`--agent codex`, `cursor`, `gemini` or another of the 13. In Claude Code, the plugin can take init's place (one or

@LoopGlitch26

Copy link
Copy Markdown
Member

Thanks for the README counts. Copilot CLI also belongs in the integration list under the badge, next to the other coding agents, and the badge then counts 35. tests/test_readme.py checks that the two agree. Two lines in README.md:

L156, add GitHub Copilot CLI after Devin:

CLI, Cursor, Gemini CLI, OpenCode, Kilo Code, goose, Qwen Code, Factory Droid, Antigravity CLI, Crush, Devin, GitHub Copilot CLI, CrewAI, Browser Use, Pydantic AI, Google ADK, Haystack, LlamaIndex, Microsoft Agent

L12, the badge and its alt text from 34 to 35:

<a href="#add-it-to-your-agent"><img src="https://img.shields.io/badge/works_with-35_agents_%26_frameworks-F3B23B?style=flat-square&labelColor=1C1A1C" alt="Works with 35 agents and frameworks"></a>

The third change is a one-click suggestion on L241: it keeps the list item's three-space indent.

@LoopGlitch26

Copy link
Copy Markdown
Member

#605 has merged, so please merge main into this branch. It changed tracekit/policy2/ and the coding pack, which is now 2026.11-v2-80: set this branch's coding pack version to 2026.11-v2-81. If another PR takes that number first, I'll move yours on merge.

Signed-off-by: Zarif Hossain <hossainzarif2002@gmail.com>
Signed-off-by: Zarif Hossain <hossainzarif2002@gmail.com>
@LoopGlitch26

Copy link
Copy Markdown
Member

Thanks for the merge and the README line. CI fails on one test on every runner, tests/test_eval_docs.py::test_the_docs_quote_the_results: the branch's eval/results/e21_policy_robustness.json now carries latency numbers and a machine line from your machine, and docs/evaluation.md quotes main's.

Please take main's version of that file and change only the two policy_hash lines the coding pack's new version moves, as #732 did. The latency section stays as recorded on main; the offline evals job will say if a hash is off.

@LoopGlitch26

Copy link
Copy Markdown
Member

I made the last changes myself and carried this in #782, since a push to your branch was refused. Your eleven commits are in it unchanged, with your authorship and sign-offs. On top of them:

  1. A merge of main, which now has Cline's hooks (Coding agents: hooks for Cline, CLI and VS Code extension (#695) #732) in the same places: both harnesses are kept, TK-F009 and TK-F015 read both harnesses' folders, and the README counts 36 and "another of the 14".
  2. The coding pack is 2026.11-v2-84 (main is at -81 and two open PRs take the next numbers; the final number is settled at merge). The three results files are main's with only the new policy hashes, so test_the_docs_quote_the_results passes again.

This PR closes when #782 merges.

@LoopGlitch26
LoopGlitch26 merged commit 2dd55e1 into Cygnux-Labs:main Nov 13, 2026
27 of 34 checks passed
@LoopGlitch26

Copy link
Copy Markdown
Member

Merged with your commits in #799 (via #782): Tracekit now guards GitHub Copilot CLI, including its typed shell input and its policy folder. It ships in the next release. Thank you.

And a follow-up to the earlier request to run Tracekit for a week: if you've been running it on your coding agent, tracekit report --share prints counts only (no commands, paths or content). Posting it with the field-report form (https://github.com/Cygnux-Labs/Tracekit/issues/new?template=field-report.yml) helps us more than anything else. If you haven't tried it yet, pip install -U tracekit-ai then tracekit audit --since 7d shows, read-only, what it would have held last week.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Coding agents: hooks for GitHub Copilot CLI

2 participants