Repository navigation
Policy merge train 1: eleven coding-pack PRs, one rule-id settle, evals re-recorded once - #799
Merged
Merged
Conversation
Signed-off-by: Sourav Chandhok <souravchandhok01@gmail.com>
Signed-off-by: Kausthubh <26836237+kausthubhk@users.noreply.github.com>
Signed-off-by: Zarif Hossain <hossainzarif2002@gmail.com>
Signed-off-by: Zarif Hossain <hossainzarif2002@gmail.com>
Signed-off-by: Kausthubh <26836237+kausthubhk@users.noreply.github.com>
Signed-off-by: Zarif Hossain <hossainzarif2002@gmail.com>
Signed-off-by: Zarif Hossain <hossainzarif2002@gmail.com>
Signed-off-by: Kausthubh <26836237+kausthubhk@users.noreply.github.com>
Signed-off-by: Kausthubh <26836237+kausthubhk@users.noreply.github.com>
Signed-off-by: Kausthubh <26836237+kausthubhk@users.noreply.github.com>
Signed-off-by: Zarif Hossain <hossainzarif2002@gmail.com>
Signed-off-by: Zarif Hossain <hossainzarif2002@gmail.com>
Signed-off-by: Zarif Hossain <hossainzarif2002@gmail.com>
OpenHands sends Claude Code's hook payload under its own keys (event_type,
working_dir, message), with no call id and metadata {}. terminal is Bash
(is_input: text typed into the running command, later), file_editor is Read
(view), Write (create) or Edit on path, apply_patch is Edit on every file its
headers name, glob and grep are Glob and Grep, MCP tools keep their names.
tool_input's kind is dropped, and an Observation with is_error is a
PostToolUseFailure. Calls are paired with _paired_ids. A file_editor with no
path or an apply_patch with no file header is denied in the hook.
A held call nobody answers is denied within APPROVAL_WAIT_S, under the 600 s
PreToolUse timeout, since OpenHands runs a call whose hook times out.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: aj-codespy <ajayush2301@gmail.com>
tracekit init --dev [--v2] --agent openhands merges Tracekit's hooks into $OH_PERSISTENCE_DIR/hooks.json, else ~/.openhands/hooks.json (--project: ./.openhands/hooks.json), with a backup: PreToolUse, PostToolUse, UserPromptSubmit, SessionStart and SessionEnd, matcher *, PreToolUse at 600 s and the rest at 30 s, each command wrapped by install._blocking. On Windows the command keeps cmd.exe's form (OpenHands runs it with shell=True). doctor warns (D-OPENHANDS-PROJECT) when the working directory has a .openhands/hooks.json without Tracekit's hooks, which OpenHands reads in place of the user's, and (D-OPENHANDS-CONCURRENCY) when agent_settings.json sets tool_concurrency_limit above 1, which can mis-pair identical calls. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: aj-codespy <ajayush2301@gmail.com>
TK-F009 and TK-F015 flag writes and deletes under .openhands/ (hooks.json, agent_settings.json), in a project or the home folder; coding pack 2026.10-v2-79. E17/E21 results carry the new policy hashes (decisions unchanged). docs: an OpenHands paragraph and column in coding-agents.md, the two doctor checks in doctor.md, and three limits rows: no call id (a tool_concurrency_limit above 1 can mis-pair identical calls), a project's .openhands/hooks.json replaces the user's (a repo can switch Tracekit off; remedy: doctor in the repo, --project or system mode), and Docker sandboxes can't reach the signer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: aj-codespy <ajayush2301@gmail.com>
…as (#563) From the review: the OpenHands CLI (1.16.0, its last release, on SDK 1.28.1) reads only <working dir>/.openhands/hooks.json and ~/.openhands/hooks.json; $OH_PERSISTENCE_DIR is honoured on SDK main only, so init no longer follows it. The SDK never reads hooks.json by itself (an app passes hook_config=), and Agent Canvas merges project and user hooks and runs locally by default: the docs, the limits rows and D-OPENHANDS-PROJECT's message say so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: aj-codespy <ajayush2301@gmail.com>
…d form (#563) OpenHands runs a hook with subprocess.run(shell=True), cmd.exe on Windows, so init keeps the quoted interpreter path there, as for Cursor, Droid, Crush and Devin; the installer test's PowerShell check now skips it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: aj-codespy <ajayush2301@gmail.com>
Must fix: - task, task_tool_set and delegate are denied in the hook: they run a sub-agent with hook_config None, whose calls would go undecided and unrecorded. - The terminal is interactive (bash in tmux): an is_input call, a key name it sends as a keystroke (ENTER, TAB, UP, C-x, ...) and a line left open by a trailing backslash are marked typed, and the engine holds a shell call marked typed (TK-SHELL-PARSE, cause typed). The mark only adds a hold; the parser already holds an unclosed quote or heredoc. Smaller: - The gemini preset's write_file, edit, read_file and list_directory map to Write, Edit, Read and LS. - init merges into hooks.json in whichever of OpenHands' three forms it is written (the hooks wrapper, the direct map, HookConfig.save's snake_case fields), and doctor reads all three. - D-OPENHANDS-CONCURRENCY is removed: results come back in call order. The limits row names the real mis-pairing: a call rejected in the CLI's prompt or failed by its tool runs no PostToolUse. - CLI 1.16.0 as released is on SDK 1.21.0, not 1.28.1. - A limits row for Windows: the hook command isn't wrapped, and the PowerShell terminal is decided as Bash. - D-OPENHANDS-PROJECT counts only a v2 PreToolUse hook, and reads the project file from $OPENHANDS_WORK_DIR. - An MCP call is decided on its data (1.21.0's MCPToolAction shape), and the recorded case has that shape. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: aj-codespy <ajayush2301@gmail.com>
Signed-off-by: sourav chandhok <souravchandhok01@gmail.com>
Signed-off-by: sourav chandhok <souravchandhok01@gmail.com>
Signed-off-by: Zarif Hossain <hossainzarif2002@gmail.com>
Signed-off-by: Zarif Hossain <hossainzarif2002@gmail.com>
Signed-off-by: sourav chandhok <souravchandhok01@gmail.com>
Signed-off-by: Kausthubh <26836237+kausthubhk@users.noreply.github.com>
tracekit init --agent codebuddy (--project, --v2, uninstall, doctor) writes Tracekit's hooks into ~/.codebuddy/settings.json ($CODEBUDDY_CONFIG_DIR when set) or a project's .codebuddy/settings.json: PreToolUse, PostToolUse, PostToolUseFailure, UserPromptSubmit, SessionStart and SessionEnd, 600 s for PreToolUse. CodeBuddy sends Claude Code's payloads and tool names, so the hook reuses that pipeline and blocks with exit 2 plus its permissionDecision deny. - NotebookRead and ViewImage are a Read of the path each reads - DeferExecuteTool is decided as the call it runs; the deferred tool's own PreToolUse decides it again under the same id, and the signer supersedes the decision the call doesn't complete, so no decided_not_completed gap - LibraryUpload is decided as an upload: TK-D022 denies a credentials file (the twin of TK-D006 and TK-D011), TK-F024 flags every upload as egress (held on the strict pack); a file: URI is read as the file it names - TK-F009 and TK-F015 hold .codebuddy/ and judge a CodeBuddy worktree's files as they judge Claude Code's .claude/worktrees/ - coding pack 2026.11-v2-82, strict pack 2026.11-strict-11; eval results change only in their policy hashes - the plugin's setup skill from tracekit-skills' skills/cline-codebuddy, its blob id re-pinned Payloads in tests/data/harness/codebuddy.json are derived from CodeBuddy Code 2.183.0's bundled code, not recorded in a live session (docs/limits.md). Fixes #757 Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
- LibraryUpload: file:// and file: are taken off as CodeBuddy does
(PathUtils.resolveFilePathToWorkDir parses no URL), and every form it may
open is decided: the rest, without a leading @, and percent-decoded too.
file://.env, file:@.ssh/config, file:///w/#/../.env and the like are denied
on every pack, at least as strictly as a shell upload of the file and as
main's TK-S004 read the unmapped call
- a deferred call the policy holds can't run (its approval is consumed by the
first PreToolUse); a {content, error} result completes it as an error, not
as a call that ran; limits row with CodeBuddy's deferToolLoading fix
- doctor names a CodeBuddy settings file with a PreToolUse hook that runs
after Tracekit's (the last permissionDecision stands); limits row
- TK-D022 reads a key's name up to NAME_MAX (255)
- init --project follows $CODEBUDDY_PROJECT_CONFIG_DIR, and TK-F015 holds a
write through it by name; limits row for dataLayout.configDir
- limits row: a v1 deferred call is recorded twice
- tests for doctor.USER_ENV's CODEBUDDY_CONFIG_DIR and said_since's skip
- README reverted: its agent lists and badge wait for the owner's review
Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
Signed-off-by: Kausthubh <26836237+kausthubhk@users.noreply.github.com>
Conflicts, all kept both sides (gptme from #778, OpenHands from #662): AGENT_NAMES, tool(), normalise()'s Claude Code payload list, harness_hooks' docstring and said_since list; coding.yaml's harness_config patterns (TK-F009, TK-F015: #778's gptme paths and #662's .openhands); the docs/coding-agents.md section and table (a gptme and an OpenHands column). README's --agent line, coding.yaml's version and the E17/E21 results: kept this side's (the settle commit sets them). Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
Conflicts, all kept both sides (gptme and OpenHands above, Copilot CLI here): AGENT_NAMES, config_path's table, harness_hooks' docstring and said_since list; coding.yaml's harness_config patterns (TK-F009, TK-F015: .copilot, .github/hooks, .github/copilot/settings*.json and /etc/github-copilot beside .openhands and gptme's paths); the installer tests (an OpenHands and the Copilot tests; the JSON-config list names both); docs/coding-agents.md (init lines, the v1 list, the Copilot section after OpenHands', the deny-display rows), docs/doctor.md (D-GPTME-LOAD and #782's D-HOOKS-TIMEOUT row), README's harness list. README's --agent line, coding.yaml's version and the E17/E21 results: kept this side's (the settle commit sets them). Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
Conflicts, all kept both sides (gptme, OpenHands and Copilot CLI above, CodeBuddy Code here): agent_hooks' docstring, tool names, AGENT_NAMES, tool() (Copilot's and CodeBuddy's branches), normalise()'s Claude Code payload list, _reply (Copilot's JSON deny; CodeBuddy with Codex and Droid), _command's Windows list (OpenHands and CodeBuddy keep _hook_command's form); harness_hooks' docstring and said_since list; doctor's USER_ENV (COPILOT_HOME and CODEBUDDY_CONFIG_DIR); coding.yaml's harness_config patterns and their comment (TK-F009, TK-F015); the hook tests (OpenHands' and CodeBuddy's v2 tests, the Copilot MCP and CodeBuddy deferred-call classes, the installer lists and counts); a duplicated Engine import both sides added to test_harness_hooks_v2.py is kept once; docs/coding-agents.md (init lines, the v1 list, the CodeBuddy section after Copilot's, a CodeBuddy column, the deny-display rows, agent names), docs/doctor.md (D-HOOKS-EVENT names Copilot CLI and CodeBuddy Code; D-HOOKS-ORDER), docs/integrations.md (both rows), README's harness list. README's --agent line, coding.yaml's version and the E17/E21 results: kept this side's (the settle commit sets them). Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
…as edits) into the policy train Conflicts: docs/coding-agents.md's table (#767's Cursor tool names cell merged into the table with the gptme, OpenHands and CodeBuddy Code columns); coding.yaml's version and the E17/E21 results (kept this side's; the settle commit sets them). Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
Rule ids: #780's TK-A063 to TK-A066 become TK-A071 to TK-A074, and #788's TK-A067 to TK-A070 become TK-A075 to TK-A078 (#778 keeps TK-A063 to TK-A065 and #787 TK-A067 and TK-A068; TK-A066, TK-A069 and TK-A070 stay unused, so no id names two rules across the members), in the coding pack, its tests and docs. Packs: coding 2026.11-v2-82, strict 2026.11-strict-11 and dev 2026.11-dev-5, one step above main's each. dev maps Claude Code's Workflow to `harness` (TK-A062 held it as an unmapped tool given a script), with a benign test. Harness lists: gptme, OpenHands, Copilot CLI, CodeBuddy Code and Cline in AGENTS.md, README (badge 39, the --agent list, "another of the 17"), docs/coding-agents.md's init lines, docs/doctor.md, docs/integrations.md and docs/support.md; gptme's paths in TK-F009's comment. Docs: a limits row for #788's known gaps; two bare | in #781's limits row escaped. The plugin's SKILL.md is tracekit-skills 4bda57a. Evals re-recorded once: E17 (901 harmful stopped, 1 false alarm in 944), E21 policy robustness (held-out 279 / 420 default, 305 / 420 strict; latency rows kept as main's) and E21 robustness (901 originals and 2,701 rewrites contained). Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
The settle commit split strict's E17 count across two sentences; tests/test_eval_docs.py reads it as one quote. Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
This was referenced Nov 13, 2026
Merged
…it is the same call #761's block() completed the decision an earlier PreToolUse of the same call id left open, whatever that call's tool and arguments. A different call reusing an open id (a model that writes its own ids) then signed the first call's result as an error with the block's text and dropped its real result. _not_run now takes this call's digest and leaves another call's decision for its own PostToolUse. CodeBuddy's deferred call (same tool and arguments) is unchanged. Also from the train's review: the TK-D014 limits row names a subfolder followed by .. (/etc/ssh/..), and dev.yaml's comment names Workflow with the harness tools. Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The first policy merge train. Eleven reviewed pull requests change the coding pack, and each merge made the others' recorded E17 and E21 results stale. This branch merges each member's reviewed head as it is, first-time contributors first. One commit then settles where they collide: rule ids, pack versions, harness lists and the evals, re-recorded once.
Merge it with a merge commit (
gh pr merge 799 --repo Cygnux-Labs/Tracekit --merge --match-head-commit c25a76434a4c902a0e77813a7eeb793ec7fc74f1), not squash or rebase. Then every member's head is in main, and GitHub shows each member PR as merged.Members, in merge order
5b2fc1d596b859b1966e72fbde035173085cb4e6.pthin site-packages held as a.pthfile (TK-A067, TK-A068)6a9f1dab1b9c7ac3dfb7600c210ebaeef8dc396bfb189e9d08da70988035dbb1fccced92fb34c03ddca9814b2217ce729a1a30e4c838e4dcb5f7517ed6cb17a65cad7fe0c887f816a66ffd07f7fe0c77d0ac52c38affa9261ed88f0e6a7da6872577cf2b2f2907ce98a5a01319fbbd29ce4b53cd2f09cc05db340466a1da0e90da892a46e703becf6350ce64351065a4aa503f5060e7152d83451d5f0512069fdd1aa46fd35f692031cb1b4fc81ffa66b3c141d6c75778cb59506e6c980d0be8e0ba4dc941c46141Each member is merged with
git merge --no-ff --signoff. No member commit is rewritten. Each merge commit's message names its conflicts and how they were settled: every conflict kept both sides. The branch then merges main once more, for #786 and #784, with no conflicts.The settle commit is
18aa79fd2b.Rule ids
#778 keeps TK-A063 to TK-A065, and its v1
default.yamlkeeps TK-A064 and TK-A065. #787 keeps TK-A067 and TK-A068. #781's TK-A061, #783's TK-D014 and #761's TK-D022 and TK-F024 collide with nothing. TK-A066, TK-A069 and TK-A070 stay unused, so no id names two different rules across the members. Every reference moved with its rule: the pack, the tests,docs/policy-v2.mdanddocs/limits.md.Pack versions
Each is one step above main's:
2026.11-v2-81→2026.11-v2-822026.11-strict-10→2026.11-strict-112026.11-dev-4→2026.11-dev-5default.yaml:2026.10-2→2026.10-3Also in the settle commit
Workflowtoharness. Before, TK-A062 held it as an unmapped tool given a script. The new benign test fails without the mapping.said_since(both in the merges)--agentlist, "another of the 17"docs/coding-agents.md's init lines and v1 listdocs/doctor.md's D-HOOKS-EVENT and D-HOOKS-TIMEOUTdocs/integrations.mdanddocs/support.mddocs/limits.mdgets a row for Policy: hold logins, setuid bits, cron spool writes and startup-file reverse shells (#750; carries #752) #788's known gaps. Two bare|in Policy: hold CI workflow writes that upload secrets (TK-A061) (carries #663) #781's row are escaped; the table test failed on them.4bda57a. Merge tracekit-setup: the cline and codebuddy agents tracekit-skills#2 first.docker runand three startup-file edits. The latency rows stay main's, as the convention says.docs/evaluation.md's "over target" sentence now gives these numbers.Decisions against main
Main's corpora were decided on main and on this head:
Results:
Gates
Run on macOS arm64 against
a29baa4d6a, the settle commit with main merged in:make test-ts lint build test:tests/test_eval_docs.py: the settle commit had split strict's E17 count across two sentences.c25a764joins them, andtests/test_eval_docs.pyandtests/test_docs.pypass on this head (13 passed). The full suite was not run again for that one-paragraph doc change.Linux root-only and
/proctests skip on a laptop. CI's Linux run is the gate for those.The CI evals job, run locally on this head:
e14_outage --quickpassed.samefilters for e4 v2, E17, E21 robustness and E21 policy robustness show no difference.