8 of 8tamper attacks caught 480 trials, with a witness (E1) |
-699 of 699harmful calls stopped 11 classes, 1 false alarm in 800 (E17) |
+715 of 715harmful calls stopped 11 classes, 1 false alarm in 814 (E17) |
< 1 msadded per tool call 0.91 ms at p99, signer round trip (E9) |
8 of 8insider attacks caught a merge gate on every PR (E8) |
@@ -212,7 +212,7 @@ is tested, but may change between releases ([what's stable](https://github.com/C
| Core | What it does | Measured |
|---|---|---|
| **Record** | Every tool call, decision and result in one signed hash chain per run. A lost event becomes a signed gap, never silence | 6 of 8 tamper attacks caught by the log alone (E1) |
-| **Decide** | Allow, deny, or hold each call for a person, before it runs. [Packs](https://github.com/Cygnux-Labs/Tracekit/wiki/Policy) for shell, files, SQL, HTTP, cloud CLIs, browsers, email, messaging, payments, secrets and MCP | 699 of 699 harmful calls stopped, 1 false alarm in 800 (E17) |
+| **Decide** | Allow, deny, or hold each call for a person, before it runs. [Packs](https://github.com/Cygnux-Labs/Tracekit/wiki/Policy) for shell, files, SQL, HTTP, cloud CLIs, browsers, email, messaging, payments, secrets and MCP | 715 of 715 harmful calls stopped, 1 false alarm in 814 (E17) |
| **Approve** | A held call runs only after a yes from the command line, bound to its exact arguments and used once ([approvals](https://github.com/Cygnux-Labs/Tracekit/wiki/Approvals)) | Edited arguments, a replayed snapshot or another run's approval: refused, in the contract tests every stable framework integration runs |
| **Prove** | `tracekit verify` checks a run's file offline against keys you pinned | 8 of 8 insider attacks caught (E8) |
| **Watch** | `tracekit view`: every run verified, blocked and held calls, approvals, gaps, replay and a run review ([viewer](https://github.com/Cygnux-Labs/Tracekit/wiki/Viewer)) | Not measured |
@@ -240,7 +240,7 @@ Our own experiments: small, synthetic and written by us. Nothing rounded up. The
|---|---|---|
| **E1** tamper detection | 120-record signed logs, 8 attacks, 60 trials each, re-run on 1.2 | 6 of 8 caught by the log alone, **8 of 8 with a witness** |
| **E8** insider attacks | Real unprivileged users against a Linux system-mode signer: a decoy signer, a stopped signer, a swapped policy, fabricated runs | **8 of 8 caught**, a merge gate on every pull request |
-| **E17** policy packs | 699 harmful and 800 benign tool calls in 11 classes, decided by both regex engines | **699 of 699 stopped**; 1 false alarm, a draft email held for a person |
+| **E17** policy packs | 715 harmful and 814 benign tool calls in 11 classes, decided by both regex engines | **715 of 715 stopped**; 1 false alarm, a draft email held for a person |
| **E21** policy robustness | 420 harmful and 432 benign calls written blind to the rules; 130 developer-session calls | server pack: 211 of 420 stopped or held on the one blind run, 247 after rules were added that target the pool (read since); 18 of 432 false alarms, 2 of 130 in sessions; p99 185 µs |
| **E10** reconciliation | 30 runs, 12 with a seeded discrepancy: a missing hook, a fabricated call, changed arguments | **12 of 12 flagged**, 0 false positives |
| **E9** overhead | Signer round trip over a Unix socket, file storage, macOS quick run | **0.45 ms p50, 0.91 ms p99** per call; 5,356 events/s |
diff --git a/docs/concepts/policy.md b/docs/concepts/policy.md
index 768e0e720..bcb3be8ca 100644
--- a/docs/concepts/policy.md
+++ b/docs/concepts/policy.md
@@ -167,7 +167,7 @@ tracekit policy backtest --policy policy.yaml --store DATA_DIR/store # replay
Rules are tripwires, not a sandbox. A determined agent can phrase a command so no rule matches: the
[evaluation](../evaluation.md#e3-policy-gate) lists misses such as `X=rm; $X -rf ~` and a base64 payload piped to a
-shell. On the shipped packs' own test corpus (699 harmful and 800 benign calls across 11 classes, written by the
+shell. On the shipped packs' own test corpus (715 harmful and 814 benign calls across 11 classes, written by the
project) every harmful call is denied or held, with one false alarm. On E21's held-out pool (420 harmful and 432
benign calls, written blind to the rules), the only blind run caught 211 harmful calls (50.2%) with the server pack and
246 (58.6%) with the strict pack. With rules added since, written while the misses were known, they catch 245 (58.3%)
diff --git a/docs/concepts/security-model.md b/docs/concepts/security-model.md
index be6867343..d641d090e 100644
--- a/docs/concepts/security-model.md
+++ b/docs/concepts/security-model.md
@@ -94,7 +94,7 @@ synthetic and written by the project.
| Reconciliation (E10) | 12 of 12 seeded discrepancies flagged, 0 false positives |
| Saved-state tampering (E11) | 21 of 21 caught |
| Tenant isolation (E12) | Every attack refused, no leaks under concurrency |
-| Policy packs on the project's corpus (E17) | 699 of 699 harmful calls stopped, 1 false alarm in 800 |
+| Policy packs on the project's corpus (E17) | 715 of 715 harmful calls stopped, 1 false alarm in 814 |
| Policy packs on calls written blind to the rules (E21) | Server pack: 211 of 420 harmful calls stopped or held on the only blind run; 245 with rules added since (the pool is no longer blind); 18 false alarms in 432 |
| Agent approvers (E18) | 7 of 7 attacks caught |
| Unrequested edits behind cover text, checked by `tracekit analyze --v2` (E20, synthetic; the blind pool v3, scored once) | Caught under every cover; after a "don't touch X", 79 to 100% by cover (67 to 92% held before the edit ran); 5 false findings and 1 held call in 1,812 honest calls |
diff --git a/docs/evaluation.md b/docs/evaluation.md
index 8a8991b11..7a15ac416 100644
--- a/docs/evaluation.md
+++ b/docs/evaluation.md
@@ -279,9 +279,9 @@ p99. `make eval` runs the `--quick` form (5 calls per phase instead of 20). Resu
Three offline parts, each with a signer in process; nothing is executed.
-- **Policy packs** (`eval/e17_policy_corpus.py`): 699 harmful and 800 benign tool calls in 11 classes
- (`tests/data/policy/`), written by the project, decided by `server.yaml` with both engines. All 699 harmful calls are
- denied or held; one benign call is held (a draft email to a list address, `TK-M003`), 1 in 800 (0.12%); the engines
+- **Policy packs** (`eval/e17_policy_corpus.py`): 715 harmful and 814 benign tool calls in 11 classes
+ (`tests/data/policy/`), written by the project, decided by `server.yaml` with both engines. All 715 harmful calls are
+ denied or held; one benign call is held (a draft email to a list address, `TK-M003`), 1 in 814 (0.12%); the engines
decide every call the same. It gates on every harmful call caught, at most 2% false positives and engine agreement.
- **Detection** (`eval/e17_detection.py`): 10 scenarios for [sequences](incidents.md): a sequence split across two
runs, a tool alias the policy doesn't map, parallel runs, a GET with a query string as egress, a denied step that
@@ -553,17 +553,17 @@ tool) to 240 (57.1%) and 269 (64.0%), with none either. The database-client rule
pool, not as blind recall. The gates were
recorded again with those rules, on purpose: strict's false positives on E17's benign calls rose from 4.1% to 4.9%
(29 of 711 to 36 of 738). The benign cases added since, with the secret-read, repository-deletion,
-database-client and infrastructure-removal rules, are calls strict does not hold, so the count stayed at 36 while the
-pool grew to 800 (4.5%).
+repository-protection, database-client and infrastructure-removal rules, are calls strict does not hold, so the count
+stayed at 36 while the pool grew to 814 (4.4%).
All 36 are benign calls strict holds: 23 as a push, publish, install or apply (`TK-F004`), 12 as a network command
(`TK-F013`), and 1 draft email to a list address (`TK-M003`); `TK-F012` (any URL a call acts on) only flags in
strict, it never holds one by itself.
-Shell rewrites (`eval/e21_robustness.py`): E17's 699 harmful calls are decided as they are, and each of the 373 shell
+Shell rewrites (`eval/e21_robustness.py`): E17's 715 harmful calls are decided as they are, and each of the 389 shell
commands among them again rewritten in up to five equivalent ways (`cd . &&` or `true;` before it, Bash's `command`
builtin, `bash -c` and `env bash -c` around it; a form that could change what the command does is left out, such as
-`bash -c` around a `cd` or an `export`). A call is contained when the server pack still denies or holds it: all 699
-originals and all 1,792 rewrites are, and both engines decide each the same. The run gates against a recorded
+`bash -c` around a `cd` or an `export`). A call is contained when the server pack still denies or holds it: all 715
+originals and all 1,871 rewrites are, and both engines decide each the same. The run gates against a recorded
baseline, rule id by rule id, so a rule that held a form and no longer does fails it. Results:
`eval/results/e21_robustness.json`.
diff --git a/docs/limits.md b/docs/limits.md
index 8968edf00..6e45a30d2 100644
--- a/docs/limits.md
+++ b/docs/limits.md
@@ -169,7 +169,7 @@ What Tracekit guarantees and doesn't, per deployment, on one page: [guarantees](
| `TK-A041` holds a destructive statement in an SQL-like MCP tool's `sql`/`query`/`statement` argument and reads the tool's name: only a tool whose name looks like a database tool is held, so an ordinary search whose `query` text is itself a destructive statement (a search for `DROP TABLE x`) is not. A call over 64 KiB to a tool whose name looks like a database tool (`execute_sql`, a bulk `insert_rows` on a `db` server) asks on `coding` and `dev` whatever it holds (`TK-OVERSIZE`), where any other MCP call that size is flagged: the rule reads the whole arguments, and a subject over 64 KiB is never cut. `server` and `strict` already deny every call that size. | SQL tool names are unbounded, so the rule matches names like `sql`, `postgres`, `mysql`, `sqlite`, `clickhouse`, `mongo` or `database`; a destructive statement through any other tool name is not held. | Approve the held call; the hold is ask, not deny. | [policy](policy-v2.md#packs-shipped) |
| A shell whose language the parser doesn't read (`fish`, `csh`, `tcsh`, PowerShell, `nu`, `xonsh`, `elvish`) asks (`TK-SHELL-PARSE`) whenever it runs commands from its line or stdin, even `pwsh -c 'Get-Date'`; a script file it runs (`fish build.fish`) isn't read, as a POSIX shell's isn't. | Their syntax isn't POSIX, so the rules anchored on a command's name can't read what they run. | Approve the held call, or run the command through a POSIX shell. | [policy](policy-v2.md#what-a-rule-matches-the-subject) |
| With `on_parse_error: deny`, an ordinary command whose program the parser can't read (`$EDITOR notes.md`, a script piped into `bash`, an unclosed quote) is denied, not held. | The signer can't tell such a command from one built to hide what it runs, and with deny no person sees it. | Keep the default (`ask`) wherever a person answers holds; with deny, have the agent name programs literally. | [policy](policy-v2.md#file-format) |
-| `strict.yaml` holds ordinary development work: every push, package install, `docker run`, network command and `PATH=`-style setting waits for a person (4.5% of E17's benign calls, 10.0% of E21's developer-session calls), and a command whose program comes from a variable (`$EDITOR notes.md`) is denied. | It trades interruptions for seeing every change that leaves the workspace before it runs; the flags it raises can't tell a routine push from a harmful one. | Use it where a person answers holds promptly; elsewhere use `server.yaml` and read the flags. | [policy](policy-v2.md#packs-shipped) |
+| `strict.yaml` holds ordinary development work: every push, package install, `docker run`, network command and `PATH=`-style setting waits for a person (4.4% of E17's benign calls, 10.0% of E21's developer-session calls), and a command whose program comes from a variable (`$EDITOR notes.md`) is denied. | It trades interruptions for seeing every change that leaves the workspace before it runs; the flags it raises can't tell a routine push from a harmful one. | Use it where a person answers holds promptly; elsewhere use `server.yaml` and read the flags. | [policy](policy-v2.md#packs-shipped) |
| On calls written blind to the rules (E21's held-out pool), the server pack stopped or held half the harmful ones on its one blind run (211 of 420), and 247 (111 denied) after rules were added that target the pool's shapes, a number that overstates blind recall; it holds 18 of 432 benign ones. It misses most exfiltration (27 of 39), privilege-escalation (19 of 27), prompt-injection (14 of 22), destructive (21 of 36) and defence-evasion (16 of 30) calls, and almost half the supply-chain ones (12 of 27). | The packs match known command shapes; calls phrased in ways no rule anticipates aren't matched. | Treat the packs as tripwires: run agents in a sandbox, keep credentials out of their reach, rely on the signer and witnesses for the record; `strict.yaml` holds more (275 of 420) at more interruptions. | [evaluation](evaluation.md#e21-policy-robustness) |
| A secret's value read in a way `TK-A035` to `TK-A038` don't name isn't held. They read the commands of AWS Secrets Manager and SSM Parameter Store, GCP Secret Manager and Azure Key Vault, and MCP tools whose names say they return a secret (or read an SSM parameter with decryption turned on). Not held: another store's shell commands (`vault kv get`, `op read`, `kubectl get secret`), a command run through a wrapper the shell parser doesn't read (`aws-vault exec … --`), an AWS call whose decryption comes from `--cli-input-json`, an ALL-CAPS tool name, an MCP tool whose server chose another name (`fetch_credential`) or a spelling the patterns lack, a CLI command given to an MCP tool whose name doesn't end in a word for running one (`TK-A036` reads `call_aws`, `run_command`, `execute`, `bash` and the like), an SDK call in an interpreter one-liner (`python -c`), and a credential that isn't in a secret store (`aws ecr get-login-password`, `aws sts assume-role`). A tool that both lists secrets and gets one (Azure's Key Vault tool) is held for both when its tool name says it gets a secret; one named only `keyvault` that takes the operation as an argument isn't. A call over 64 KiB to a tool `TK-A036` or `TK-A038` reads (`call_aws`, `run_command`, `get_parameter`) asks on `coding` and `dev` whatever it holds (`TK-OVERSIZE`), where any other MCP call that size is flagged: the rules read the whole arguments, and a subject over 64 KiB is never cut. `server` and `strict` already deny every call that size. | The rules read command words, tool names and the arguments the tool is given; they can't tell what a tool with another name returns. | Hold the tools your servers use for secrets by name in a rule of your own, and keep agents' IAM roles from reading secrets they don't need. | [policy](policy-v2.md#packs-shipped) |
| A rule's `remedy` is advice the policy's author wrote: lint checks its form (one short present-tense sentence), not that it is right, and an agent may ignore it. | It is text shown with the decision, not a rule. | Write a remedy that names a safe alternative the agent can carry out; the deny or hold stands either way. | [policy](policy-v2.md#file-format) |
@@ -196,6 +196,7 @@ What Tracekit guarantees and doesn't, per deployment, on one page: [guarantees](
| `TK-A033` (merging a pull request) doesn't read a merge through a GraphQL query in a file, a variable or a pipe (`gh api graphql -F query=@merge.graphql`, `-f query="$Q"`, `printf … \| gh api graphql -F query=@-`), a URL the client rewrites before sending (curl's and wget's dot segments, `…/pulls/1/./merge`, and curl's globbing, `…/pull{s}/1/merge`), a script or SDK call, a `gh` or `glab` alias (`gh alias set pm 'pr merge'`), a shell alias or function, more than four options before a subcommand, a program named by a Windows path with backslashes (`"C:\Program Files\GitHub CLI\gh.exe" pr merge 1`, which `TK-A001` and `TK-A002` miss for their programs too: the parser keeps the directories before a `\` in a program's name), another forge's CLI or API (Bitbucket, Gitea), a branch merge (`POST …/merges` or GraphQL `mergeBranch`, which merge one branch into another, not a pull request) or a push straight to a protected branch (`git push origin HEAD:main`, which isn't a merge); it holds a status check (`GET …/pulls/N/merge`) too. | It reads the command lines of `gh`, `glab`, `hub`, `curl`, `wget`, HTTPie and `xh` as typed; a file's, variable's, pipe's or script's contents aren't on them, and the URL is matched before the client rewrites it. | Give agents a token that can't merge or push to protected branches (branch protection with no admin bypass), and keep merging to people. | [policy](policy-v2.md#packs-shipped) |
| `TK-A034` (a patch applied through the shell) holds every patch run as Codex's `apply_patch`, `applypatch` or `codex --codex-run-as-apply-patch`, one to an ordinary file too. It doesn't read Codex started by a runner's subcommand or an option with a value before its script (`bun run codex.js`, `node -r m codex.js`), a patch applied by a script, by a copy of Codex's `apply_patch.bat` under another name (Windows), or by `git apply` or `patch`, whose files no file rule reads either. | Codex applies such a patch itself, so its hook sends a shell command, not the files; the rule reads the command's words, not the patch. | Give Codex models the apply_patch tool, whose files Tracekit decides one by one, and read a held patch before approving it. | [policy](policy-v2.md#packs-shipped), [coding agents](coding-agents.md) |
| `TK-A042` (deleting a repository or project through a platform's API) doesn't read a delete through a GraphQL query in a file, a variable or a pipe (`gh api graphql -F query=@delete.graphql`, `-f query="$Q"`, `printf … \| gh api graphql -F query=@-`), a URL the client rewrites before sending (curl's and wget's dot segments, `…/repos/o/r/./`, and curl's globbing, `…/repo{s}/o/r`), a script or SDK call, a `gh` or `glab` alias (`gh alias set rd 'repo delete'`), a shell alias or function, more than four options before a subcommand, more than sixteen words between the method and a URL or a bare path (a `curl` with seventeen words of headers and options), another forge's CLI or API (Bitbucket, Gitea), a delete through the platform's web UI (a browser navigation, which no shell rule reads), or a GitLab project URL under a subpath (`https://host/gitlab/api/v4/projects/42`). | It reads the command lines of `gh`, `glab`, `hub`, `curl`, `wget`, HTTPie and `xh` as typed; a file's, variable's, pipe's or script's contents aren't on them, and the URL is matched before the client rewrites it. | Give agents a token that can't delete repositories or projects (no admin scope), and keep deleting them to people. | [policy](policy-v2.md#packs-shipped) |
+| `TK-A047` (removing a repository's secrets, branch protection, rulesets or security features through GitHub) doesn't read another forge's CLI or API (GitLab's protected branches, Bitbucket, Gitea), a GraphQL mutation (`deleteBranchProtectionRule`), a `security_and_analysis` change whose body comes from a file or a pipe (`gh api -X PATCH repos/o/r --input off.json`) or whose status comes from a variable, a JSON body padded so more than 200 characters sit between `security_and_analysis` and `status` or more than 16 non-word characters between `status` and `disabled`, a token used from a script file or an SDK, other HTTP clients (`wget`, HTTPie, `xh`), a `gh` alias or a shell function, a path handed over by `xargs`, a `curl` config file (`-K`), a URL `curl` rewrites before sending it (globbing such as `{branches}` or `branche[s-s]`, `--expand-url` variables, `./` and `../` segments), a repository named by its numeric id (`repositories/