Skip to content

Policy merge train 1: eleven coding-pack PRs, one rule-id settle, evals re-recorded once - #799

Merged
LoopGlitch26 merged 89 commits into
mainfrom
agent/policy-train-1
Nov 13, 2026
Merged

LoopGlitch26 merged 89 commits into
mainfrom
agent/policy-train-1

Conversation

@LoopGlitch26

@LoopGlitch26 LoopGlitch26 commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

The first policy merge train. Eleven reviewed pull requests change the coding pack, and each merge made the others' recorded E17 and E21 results stale. This branch merges each member's reviewed head as it is, first-time contributors first. One commit then settles where they collide: rule ids, pack versions, harness lists and the evals, re-recorded once.

Merge it with a merge commit (gh pr merge 799 --repo Cygnux-Labs/Tracekit --merge --match-head-commit c25a76434a4c902a0e77813a7eeb793ec7fc74f1), not squash or rebase. Then every member's head is in main, and GitHub shows each member PR as merged.

Members, in merge order

# PR Head Contributor What
1 #787 5b2fc1d596b859b1966e72fbde035173085cb4e6 nx9161 a .pth in site-packages held as a .pth file (TK-A067, TK-A068)
2 #778 6a9f1dab1b9c7ac3dfb7600c210ebaeef8dc396b souravC01 gptme hooks (TK-A063 to TK-A065)
3 #783 fb189e9d08da70988035dbb1fccced92fb34c03d carries Vedant-200's #762 deny deleting or locking a system folder (TK-D014)
4 #780 dca9814b2217ce729a1a30e4c838e4dcb5f7517e carries munjikesh's #731 Docker or Podman socket calls and cgroup release_agent writes
5 #781 d6cb17a65cad7fe0c887f816a66ffd07f7fe0c77 carries kausthubhk's #663 CI workflow writes that upload secrets (TK-A061); built on #780
6 #788 d0ac52c38affa9261ed88f0e6a7da6872577cf2b carries jay23606's #752 logins, setuid bits, cron spool writes and startup-file shells (and TK-A011)
7 #662 2f2907ce98a5a01319fbbd29ce4b53cd2f09cc05 aj-codespy OpenHands hooks
8 #782 db340466a1da0e90da892a46e703becf6350ce64 carries hossainzarif23's #710 GitHub Copilot CLI hooks
9 #761 351065a4aa503f5060e7152d83451d5f0512069f LoopGlitch26 CodeBuddy Code hooks (TK-D022, TK-F024)
10 #767 dd1aa46fd35f692031cb1b4fc81ffa66b3c141d6 LoopGlitch26 Cursor's StrReplace, ApplyPatch and EditNotebook decided as edits
11 #792 c75778cb59506e6c980d0be8e0ba4dc941c46141 LoopGlitch26 1.7.0 review fixes; E21 gated pool by pool

Each member is merged with git merge --no-ff --signoff. No member commit is rewritten. Each merge commit's message names its conflicts and how they were settled: every conflict kept both sides. The branch then merges main once more, for #786 and #784, with no conflicts.

The settle commit is 18aa79fd2b.

Rule ids

Member Was Now
#780 TK-A063, TK-A064, TK-A065, TK-A066 TK-A071, TK-A072, TK-A073, TK-A074
#788 TK-A067, TK-A068, TK-A069, TK-A070 TK-A075, TK-A076, TK-A077, TK-A078

#778 keeps TK-A063 to TK-A065, and its v1 default.yaml keeps TK-A064 and TK-A065. #787 keeps TK-A067 and TK-A068. #781's TK-A061, #783's TK-D014 and #761's TK-D022 and TK-F024 collide with nothing. TK-A066, TK-A069 and TK-A070 stay unused, so no id names two different rules across the members. Every reference moved with its rule: the pack, the tests, docs/policy-v2.md and docs/limits.md.

Pack versions

Each is one step above main's:

  • coding: 2026.11-v2-81 → 2026.11-v2-82
  • strict: 2026.11-strict-10 → 2026.11-strict-11
  • dev: 2026.11-dev-4 → 2026.11-dev-5
  • v1 default.yaml: 2026.10-2 → 2026.10-3

Also in the settle commit

  • dev maps Claude Code's Workflow to harness. Before, TK-A062 held it as an unmapped tool given a script. The new benign test fails without the mapping.
  • The harness lists now name gptme, OpenHands, Copilot CLI, CodeBuddy Code and Cline:
    • AGENT_NAMES and said_since (both in the merges)
    • TK-F009 and TK-F015's paths, with gptme added to TK-F009's comment
    • AGENTS.md
    • README: badge 39, the --agent list, "another of the 17"
    • docs/coding-agents.md's init lines and v1 list
    • docs/doctor.md's D-HOOKS-EVENT and D-HOOKS-TIMEOUT
    • docs/integrations.md and docs/support.md
  • docs/limits.md gets a row for Policy: hold logins, setuid bits, cron spool writes and startup-file reverse shells (#750; carries #752) #788's known gaps. Two bare | in Policy: hold CI workflow writes that upload secrets (TK-A061) (carries #663) #781's row are escaped; the table test failed on them.
  • The plugin's SKILL.md is now tracekit-skills 4bda57a. Merge tracekit-setup: the cline and codebuddy agents tracekit-skills#2 first.
  • The evals are re-recorded once:
    • E17: 901 of 901 harmful calls stopped, 1 false alarm in 944.
    • E21 policy robustness: held-out 279 / 420 on default and 305 / 420 on strict; blind 104 and 109 of 144. No new false positive on the held-out, session or blind calls. Strict's E17 false positives are 50 of 944 (5.3%): the four new ones are members' benign cases, docker run and three startup-file edits. The latency rows stay main's, as the convention says.
    • E21 robustness: all 901 originals and all 2,701 rewrites contained.
    • docs/evaluation.md's "over target" sentence now gives these numbers.

Decisions against main

Main's corpora were decided on main and on this head:

  • the corpora: E17 (1,730 calls), E21 held-out (852), the four blind pools (287), the developer sessions (130) and E21's 2,451 shell rewrites of E17's harmful calls
  • every pack (10), both engines, both through the engine's decide and through the signer's evaluate

Results:

  • 0 decisions looser than main, on any call.
  • 0 changes on any benign call, and none on E17, the rewrites or the sessions.
  • 25 held-out harmful calls are stricter on default, coding, dev and server-core. Of these, 24 are newly held: TK-A077 7, TK-A076 6, TK-D014 3, TK-A071 3, TK-A075 2, TK-A073 1, TK-A078 1, TK-A011 1. One goes from ask to deny (TK-D014).
  • 20 held-out harmful calls are stricter on strict.
  • 1 blind harmful call is newly held on every one of those packs (TK-A061, amandeavor).
  • Some held-out harmful calls keep their verdict and gain a rule id: 2 on default, coding, dev and server-core, and 7 on strict.
  • The engines agree on every decision, on main and here.

Gates

Run on macOS arm64 against a29baa4d6a, the settle commit with main merged in:

make test-ts lint build test:

  • test-ts: 162 tests, 152 passed, 10 skipped, 0 failed.
  • lint: clean.
  • build: sdist and wheel built; twine check passed.
  • test: 5,134 passed, 1 failed, 395 skipped, 10 xfailed. The failure was tests/test_eval_docs.py: the settle commit had split strict's E17 count across two sentences. c25a764 joins them, and tests/test_eval_docs.py and tests/test_docs.py pass on this head (13 passed). The full suite was not run again for that one-paragraph doc change.

Linux root-only and /proc tests skip on a laptop. CI's Linux run is the gate for those.

The CI evals job, run locally on this head:

  • All 17 evals and e14_outage --quick passed.
  • The deterministic results equal the committed files.
  • The same filters for e4 v2, E17, E21 robustness and E21 policy robustness show no difference.
  • The E17 and E21 threshold checks hold.

souravC01 and others added 30 commits November 3, 2026 22:58
Signed-off-by: Sourav Chandhok <souravchandhok01@gmail.com>
Signed-off-by: Kausthubh <26836237+kausthubhk@users.noreply.github.com>
Signed-off-by: Zarif Hossain <hossainzarif2002@gmail.com>
Signed-off-by: Zarif Hossain <hossainzarif2002@gmail.com>
Signed-off-by: Kausthubh <26836237+kausthubhk@users.noreply.github.com>
Signed-off-by: Zarif Hossain <hossainzarif2002@gmail.com>
Signed-off-by: Zarif Hossain <hossainzarif2002@gmail.com>
Signed-off-by: Kausthubh <26836237+kausthubhk@users.noreply.github.com>
Signed-off-by: Kausthubh <26836237+kausthubhk@users.noreply.github.com>
Signed-off-by: Kausthubh <26836237+kausthubhk@users.noreply.github.com>
Signed-off-by: Zarif Hossain <hossainzarif2002@gmail.com>
Signed-off-by: Zarif Hossain <hossainzarif2002@gmail.com>
Signed-off-by: Zarif Hossain <hossainzarif2002@gmail.com>
OpenHands sends Claude Code's hook payload under its own keys (event_type,
working_dir, message), with no call id and metadata {}. terminal is Bash
(is_input: text typed into the running command, later), file_editor is Read
(view), Write (create) or Edit on path, apply_patch is Edit on every file its
headers name, glob and grep are Glob and Grep, MCP tools keep their names.
tool_input's kind is dropped, and an Observation with is_error is a
PostToolUseFailure. Calls are paired with _paired_ids. A file_editor with no
path or an apply_patch with no file header is denied in the hook.

A held call nobody answers is denied within APPROVAL_WAIT_S, under the 600 s
PreToolUse timeout, since OpenHands runs a call whose hook times out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: aj-codespy <ajayush2301@gmail.com>
tracekit init --dev [--v2] --agent openhands merges Tracekit's hooks into
$OH_PERSISTENCE_DIR/hooks.json, else ~/.openhands/hooks.json (--project:
./.openhands/hooks.json), with a backup: PreToolUse, PostToolUse,
UserPromptSubmit, SessionStart and SessionEnd, matcher *, PreToolUse at
600 s and the rest at 30 s, each command wrapped by install._blocking. On
Windows the command keeps cmd.exe's form (OpenHands runs it with shell=True).

doctor warns (D-OPENHANDS-PROJECT) when the working directory has a
.openhands/hooks.json without Tracekit's hooks, which OpenHands reads in
place of the user's, and (D-OPENHANDS-CONCURRENCY) when agent_settings.json
sets tool_concurrency_limit above 1, which can mis-pair identical calls.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: aj-codespy <ajayush2301@gmail.com>
TK-F009 and TK-F015 flag writes and deletes under .openhands/ (hooks.json,
agent_settings.json), in a project or the home folder; coding pack
2026.10-v2-79. E17/E21 results carry the new policy hashes (decisions
unchanged).

docs: an OpenHands paragraph and column in coding-agents.md, the two doctor
checks in doctor.md, and three limits rows: no call id (a
tool_concurrency_limit above 1 can mis-pair identical calls), a project's
.openhands/hooks.json replaces the user's (a repo can switch Tracekit off;
remedy: doctor in the repo, --project or system mode), and Docker sandboxes
can't reach the signer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: aj-codespy <ajayush2301@gmail.com>
…as (#563)

From the review: the OpenHands CLI (1.16.0, its last release, on SDK 1.28.1)
reads only <working dir>/.openhands/hooks.json and ~/.openhands/hooks.json;
$OH_PERSISTENCE_DIR is honoured on SDK main only, so init no longer follows
it. The SDK never reads hooks.json by itself (an app passes hook_config=),
and Agent Canvas merges project and user hooks and runs locally by default:
the docs, the limits rows and D-OPENHANDS-PROJECT's message say so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: aj-codespy <ajayush2301@gmail.com>
…d form (#563)

OpenHands runs a hook with subprocess.run(shell=True), cmd.exe on Windows,
so init keeps the quoted interpreter path there, as for Cursor, Droid,
Crush and Devin; the installer test's PowerShell check now skips it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: aj-codespy <ajayush2301@gmail.com>
Must fix:
- task, task_tool_set and delegate are denied in the hook: they run a
  sub-agent with hook_config None, whose calls would go undecided and
  unrecorded.
- The terminal is interactive (bash in tmux): an is_input call, a key name
  it sends as a keystroke (ENTER, TAB, UP, C-x, ...) and a line left open
  by a trailing backslash are marked typed, and the engine holds a shell
  call marked typed (TK-SHELL-PARSE, cause typed). The mark only adds a
  hold; the parser already holds an unclosed quote or heredoc.

Smaller:
- The gemini preset's write_file, edit, read_file and list_directory map
  to Write, Edit, Read and LS.
- init merges into hooks.json in whichever of OpenHands' three forms it
  is written (the hooks wrapper, the direct map, HookConfig.save's
  snake_case fields), and doctor reads all three.
- D-OPENHANDS-CONCURRENCY is removed: results come back in call order.
  The limits row names the real mis-pairing: a call rejected in the CLI's
  prompt or failed by its tool runs no PostToolUse.
- CLI 1.16.0 as released is on SDK 1.21.0, not 1.28.1.
- A limits row for Windows: the hook command isn't wrapped, and the
  PowerShell terminal is decided as Bash.
- D-OPENHANDS-PROJECT counts only a v2 PreToolUse hook, and reads the
  project file from $OPENHANDS_WORK_DIR.
- An MCP call is decided on its data (1.21.0's MCPToolAction shape), and
  the recorded case has that shape.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: aj-codespy <ajayush2301@gmail.com>
Signed-off-by: sourav chandhok <souravchandhok01@gmail.com>
Signed-off-by: sourav chandhok <souravchandhok01@gmail.com>
Signed-off-by: Zarif Hossain <hossainzarif2002@gmail.com>
Signed-off-by: Zarif Hossain <hossainzarif2002@gmail.com>
Signed-off-by: sourav chandhok <souravchandhok01@gmail.com>
Signed-off-by: Kausthubh <26836237+kausthubhk@users.noreply.github.com>
tracekit init --agent codebuddy (--project, --v2, uninstall, doctor) writes
Tracekit's hooks into ~/.codebuddy/settings.json ($CODEBUDDY_CONFIG_DIR when
set) or a project's .codebuddy/settings.json: PreToolUse, PostToolUse,
PostToolUseFailure, UserPromptSubmit, SessionStart and SessionEnd, 600 s for
PreToolUse. CodeBuddy sends Claude Code's payloads and tool names, so the hook
reuses that pipeline and blocks with exit 2 plus its permissionDecision deny.

- NotebookRead and ViewImage are a Read of the path each reads
- DeferExecuteTool is decided as the call it runs; the deferred tool's own
  PreToolUse decides it again under the same id, and the signer supersedes the
  decision the call doesn't complete, so no decided_not_completed gap
- LibraryUpload is decided as an upload: TK-D022 denies a credentials file
  (the twin of TK-D006 and TK-D011), TK-F024 flags every upload as egress
  (held on the strict pack); a file: URI is read as the file it names
- TK-F009 and TK-F015 hold .codebuddy/ and judge a CodeBuddy worktree's files
  as they judge Claude Code's .claude/worktrees/
- coding pack 2026.11-v2-82, strict pack 2026.11-strict-11; eval results
  change only in their policy hashes
- the plugin's setup skill from tracekit-skills' skills/cline-codebuddy, its
  blob id re-pinned

Payloads in tests/data/harness/codebuddy.json are derived from CodeBuddy Code
2.183.0's bundled code, not recorded in a live session (docs/limits.md).

Fixes #757

Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
- LibraryUpload: file:// and file: are taken off as CodeBuddy does
  (PathUtils.resolveFilePathToWorkDir parses no URL), and every form it may
  open is decided: the rest, without a leading @, and percent-decoded too.
  file://.env, file:@.ssh/config, file:///w/#/../.env and the like are denied
  on every pack, at least as strictly as a shell upload of the file and as
  main's TK-S004 read the unmapped call
- a deferred call the policy holds can't run (its approval is consumed by the
  first PreToolUse); a {content, error} result completes it as an error, not
  as a call that ran; limits row with CodeBuddy's deferToolLoading fix
- doctor names a CodeBuddy settings file with a PreToolUse hook that runs
  after Tracekit's (the last permissionDecision stands); limits row
- TK-D022 reads a key's name up to NAME_MAX (255)
- init --project follows $CODEBUDDY_PROJECT_CONFIG_DIR, and TK-F015 holds a
  write through it by name; limits row for dataLayout.configDir
- limits row: a v1 deferred call is recorded twice
- tests for doctor.USER_ENV's CODEBUDDY_CONFIG_DIR and said_since's skip
- README reverted: its agent lists and badge wait for the owner's review

Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
Signed-off-by: Kausthubh <26836237+kausthubhk@users.noreply.github.com>
Conflicts, all kept both sides (gptme from #778, OpenHands from #662):
AGENT_NAMES, tool(), normalise()'s Claude Code payload list, harness_hooks'
docstring and said_since list; coding.yaml's harness_config patterns
(TK-F009, TK-F015: #778's gptme paths and #662's .openhands); the
docs/coding-agents.md section and table (a gptme and an OpenHands column).
README's --agent line, coding.yaml's version and the E17/E21 results: kept
this side's (the settle commit sets them).

Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
Conflicts, all kept both sides (gptme and OpenHands above, Copilot CLI
here): AGENT_NAMES, config_path's table, harness_hooks' docstring and
said_since list; coding.yaml's harness_config patterns (TK-F009, TK-F015:
.copilot, .github/hooks, .github/copilot/settings*.json and
/etc/github-copilot beside .openhands and gptme's paths); the installer
tests (an OpenHands and the Copilot tests; the JSON-config list names
both); docs/coding-agents.md (init lines, the v1 list, the Copilot section
after OpenHands', the deny-display rows), docs/doctor.md (D-GPTME-LOAD
and #782's D-HOOKS-TIMEOUT row), README's harness list. README's --agent
line, coding.yaml's version and the E17/E21 results: kept this side's (the
settle commit sets them).

Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
Conflicts, all kept both sides (gptme, OpenHands and Copilot CLI above,
CodeBuddy Code here): agent_hooks' docstring, tool names, AGENT_NAMES,
tool() (Copilot's and CodeBuddy's branches), normalise()'s Claude Code
payload list, _reply (Copilot's JSON deny; CodeBuddy with Codex and
Droid), _command's Windows list (OpenHands and CodeBuddy keep
_hook_command's form); harness_hooks' docstring and said_since list;
doctor's USER_ENV (COPILOT_HOME and CODEBUDDY_CONFIG_DIR); coding.yaml's
harness_config patterns and their comment (TK-F009, TK-F015); the hook
tests (OpenHands' and CodeBuddy's v2 tests, the Copilot MCP and CodeBuddy
deferred-call classes, the installer lists and counts); a duplicated
Engine import both sides added to test_harness_hooks_v2.py is kept once;
docs/coding-agents.md (init lines, the v1 list, the CodeBuddy section
after Copilot's, a CodeBuddy column, the deny-display rows, agent
names), docs/doctor.md (D-HOOKS-EVENT names Copilot CLI and CodeBuddy
Code; D-HOOKS-ORDER), docs/integrations.md (both rows), README's
harness list. README's --agent line, coding.yaml's version and the
E17/E21 results: kept this side's (the settle commit sets them).

Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
…as edits) into the policy train

Conflicts: docs/coding-agents.md's table (#767's Cursor tool names cell
merged into the table with the gptme, OpenHands and CodeBuddy Code
columns); coding.yaml's version and the E17/E21 results (kept this
side's; the settle commit sets them).

Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
Conflict: docs/policy-v2.md's "can't inspect" row (#778's gptme rules
and main's laptop-default wording, with #792's "whose arguments are over
64 KiB as JSON" and `oversize: ask`: kept both).

Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
Rule ids: #780's TK-A063 to TK-A066 become TK-A071 to TK-A074, and #788's
TK-A067 to TK-A070 become TK-A075 to TK-A078 (#778 keeps TK-A063 to
TK-A065 and #787 TK-A067 and TK-A068; TK-A066, TK-A069 and TK-A070 stay
unused, so no id names two rules across the members), in the coding
pack, its tests and docs.

Packs: coding 2026.11-v2-82, strict 2026.11-strict-11 and dev
2026.11-dev-5, one step above main's each. dev maps Claude Code's
Workflow to `harness` (TK-A062 held it as an unmapped tool given a
script), with a benign test.

Harness lists: gptme, OpenHands, Copilot CLI, CodeBuddy Code and Cline
in AGENTS.md, README (badge 39, the --agent list, "another of the 17"),
docs/coding-agents.md's init lines, docs/doctor.md, docs/integrations.md
and docs/support.md; gptme's paths in TK-F009's comment.

Docs: a limits row for #788's known gaps; two bare | in #781's limits
row escaped. The plugin's SKILL.md is tracekit-skills 4bda57a.

Evals re-recorded once: E17 (901 harmful stopped, 1 false alarm in 944),
E21 policy robustness (held-out 279 / 420 default, 305 / 420 strict;
latency rows kept as main's) and E21 robustness (901 originals and
2,701 rewrites contained).

Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
The settle commit split strict's E17 count across two sentences; tests/test_eval_docs.py reads it as one quote.

Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
…it is the same call

#761's block() completed the decision an earlier PreToolUse of the same call id left open, whatever that call's
tool and arguments. A different call reusing an open id (a model that writes its own ids) then signed the first
call's result as an error with the block's text and dropped its real result. _not_run now takes this call's digest
and leaves another call's decision for its own PostToolUse. CodeBuddy's deferred call (same tool and arguments)
is unchanged.

Also from the train's review: the TK-D014 limits row names a subfolder followed by .. (/etc/ssh/..), and dev.yaml's
comment names Workflow with the harness tools.

Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants