Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ and gives you one `.tkb` file per run that anyone can verify offline.**
<table>
<tr>
<td align="center" width="25%"><h3>8 of 8</h3>tamper attacks caught<br><sub><a href="#proof">480 trials</a>, with a witness (E1)</sub></td>
<td align="center" width="25%"><h3>806 of 806</h3>harmful calls stopped<br><sub><a href="#proof">11 classes</a>, 1 false alarm in 866 (E17)</sub></td>
<td align="center" width="25%"><h3>828 of 828</h3>harmful calls stopped<br><sub><a href="#proof">11 classes</a>, 1 false alarm in 881 (E17)</sub></td>
<td align="center" width="25%"><h3>&lt; 1 ms</h3>added per tool call<br><sub><a href="#proof">0.91 ms at p99</a>, signer round trip (E9)</sub></td>
<td align="center" width="25%"><h3>8 of 8</h3>insider attacks caught<br><sub><a href="#proof">a merge gate</a> on every PR (E8)</sub></td>
</tr>
Expand Down Expand Up @@ -221,7 +221,7 @@ is tested, but may change between releases ([what's stable](https://github.com/C
| Core | What it does | Measured |
|---|---|---|
| **Record** | Every tool call, decision and result in one signed hash chain per run. A lost event becomes a signed gap, never silence | 6 of 8 tamper attacks caught by the log alone (E1) |
| **Decide** | Allow, deny, or hold each call for a person, before it runs. [Packs](https://github.com/Cygnux-Labs/Tracekit/wiki/Policy) for shell, files, SQL, HTTP, cloud CLIs, browsers, email, messaging, payments, secrets and MCP | 806 of 806 harmful calls stopped, 1 false alarm in 866 (E17) |
| **Decide** | Allow, deny, or hold each call for a person, before it runs. [Packs](https://github.com/Cygnux-Labs/Tracekit/wiki/Policy) for shell, files, SQL, HTTP, cloud CLIs, browsers, email, messaging, payments, secrets and MCP | 828 of 828 harmful calls stopped, 1 false alarm in 881 (E17) |
| **Approve** | A held call runs only after a yes from the command line, bound to its exact arguments and used once ([approvals](https://github.com/Cygnux-Labs/Tracekit/wiki/Approvals)) | Edited arguments, a replayed snapshot or another run's approval: refused, in the contract tests every stable framework integration runs |
| **Prove** | `tracekit verify` checks a run's file offline against keys you pinned | 8 of 8 insider attacks caught (E8) |
| **Watch** | `tracekit view`: every run verified, blocked and held calls, approvals, gaps, replay and a run review ([viewer](https://github.com/Cygnux-Labs/Tracekit/wiki/Viewer)) | Not measured |
Expand Down Expand Up @@ -249,8 +249,8 @@ Our own experiments: small, synthetic and written by us. Nothing rounded up. The
|---|---|---|
| **E1** tamper detection | 120-record signed logs, 8 attacks, 60 trials each, re-run on 1.2 | 6 of 8 caught by the log alone, **8 of 8 with a witness** |
| **E8** insider attacks | Real unprivileged users against a Linux system-mode signer: a decoy signer, a stopped signer, a swapped policy, fabricated runs | **8 of 8 caught**, a merge gate on every pull request |
| **E17** policy packs | 806 harmful and 866 benign tool calls in 11 classes, decided by both regex engines | **806 of 806 stopped**; 1 false alarm, a draft email held for a person |
| **E21** policy robustness | 420 harmful and 432 benign calls written blind to the rules; 130 developer-session calls | server pack: 211 of 420 stopped or held on the one blind run, 252 after rules were added that target the pool (read since); 18 of 432 false alarms, 2 of 130 in sessions; p99 282 µs |
| **E17** policy packs | 828 harmful and 881 benign tool calls in 11 classes, decided by both regex engines | **828 of 828 stopped**; 1 false alarm, a draft email held for a person |
| **E21** policy robustness | 420 harmful and 432 benign calls written blind to the rules; 130 developer-session calls | server pack: 211 of 420 stopped or held on the one blind run, 254 after rules were added that target the pool (read since); 18 of 432 false alarms, 2 of 130 in sessions; p99 282 µs |
| **E10** reconciliation | 30 runs, 12 with a seeded discrepancy: a missing hook, a fabricated call, changed arguments | **12 of 12 flagged**, 0 false positives |
| **E9** overhead | Signer round trip over a Unix socket, file storage, macOS quick run | **0.45 ms p50, 0.91 ms p99** per call; 5,356 events/s |
| **E3** the v1 regex gate, on a rougher set | 16 harmful and 20 benign calls written after the rules | **11 of 16 caught**, 1 false alarm. Rules are tripwires, not a sandbox |
Expand Down
38 changes: 20 additions & 18 deletions docs/evaluation.md
Original file line number Diff line number Diff line change
Expand Up @@ -280,9 +280,9 @@ p99. `make eval` runs the `--quick` form (5 calls per phase instead of 20). Resu

Three offline parts, each with a signer in process; nothing is executed.

- **Policy packs** (`eval/e17_policy_corpus.py`): 806 harmful and 866 benign tool calls in 11 classes
(`tests/data/policy/`), written by the project, decided by `server.yaml` with both engines. All 806 harmful calls are
denied or held; one benign call is held (a draft email to a list address, `TK-M003`), 1 in 866 (0.12%); the engines
- **Policy packs** (`eval/e17_policy_corpus.py`): 828 harmful and 881 benign tool calls in 11 classes
(`tests/data/policy/`), written by the project, decided by `server.yaml` with both engines. All 828 harmful calls are
denied or held; one benign call is held (a draft email to a list address, `TK-M003`), 1 in 881 (0.11%); the engines
decide every call the same. It gates on every harmful call caught, at most 2% false positives and engine agreement.
- **Detection** (`eval/e17_detection.py`): 10 scenarios for [sequences](incidents.md): a sequence split across two
runs, a tool alias the policy doesn't map, parallel runs, a GET with a query string as egress, a denied step that
Expand Down Expand Up @@ -511,8 +511,8 @@ Code-like runs and a Python process during a fetch in 3 of 8; on the tuned set a
calls (#357, published 2026-10-24), and `amandeavor.jsonl`, 45 harmful and 45 benign (#401, 2026-10-25). A pool's
blind score is the one at its publication: the code and packs of the commit that merged it, the numbers its pull
request reported. Its misses were then read and turned into rule issues (#359 to #361 from the first pool, all
closed by the rules they asked for; #402 to #405 from the second, #402 closed by #468, #403 by #532 and the rest
open), so the rules written since were informed by the pools and
closed by the rules they asked for; #402 to #405 from the second, #402 closed by #468, #403 by #532, #404 by #549 and the
rest open), so the rules written since were informed by the pools and
today's score is not a blind score: it shows what those rules added, and guards them, not how the packs do on calls
nobody has seen.
- Developer sessions (`eval/e21_benign_sessions.jsonl`): 130 calls of ordinary sessions (Python, web, Go, Rust, docs,
Expand All @@ -527,12 +527,12 @@ The recorded results, measured after those rule changes on a pool that is no lon

| Profile | Held-out caught | of which denied | caught only as an unmapped tool | Held-out benign FP | Session FP |
|---|---|---|---|---|---|
| default | 252 / 420 (60.0%) | 114 (27.1%) | 44 | 18 / 432 (4.2%) | 2 / 130 (1.5%) |
| strict | 284 / 420 (67.6%) | 152 (36.2%) | 44 | 56 / 432 (13.0%) | 20 / 130 (15.4%) |
| default | 254 / 420 (60.5%) | 114 (27.1%) | 44 | 18 / 432 (4.2%) | 2 / 130 (1.5%) |
| strict | 285 / 420 (67.9%) | 152 (36.2%) | 44 | 56 / 432 (13.0%) | 20 / 130 (15.4%) |

Weakest categories (default, caught): exfiltration 12 of 39, privilege escalation 8 of 27, prompt injection 8 of 22,
destructive 16 of 36, defence evasion 15 of 30, supply chain 15 of 27; strongest: database 32 of
32, Kubernetes 23 of 27. By evasion, calls with none are caught 153 of 275; quoting 9 of 18, environment prefixes 8 of 14, substitution 40
32, Kubernetes 25 of 27. By evasion, calls with none are caught 154 of 275; quoting 9 of 18, environment prefixes 8 of 14, substitution 40
of 49, interpreter one-liners 23 of 28 (many held as unreadable, `TK-SHELL-PARSE`; a program a variable names is read,
and one that is a bare program name, `K=kubectl; $K …`, is still held: it only hides that name). Per category, severity and evasion, and every miss
and false positive: `eval/results/e21_policy_robustness.json`. The two engines decided every call the same.
Expand All @@ -544,10 +544,10 @@ denied or held:
|---|---|---|---|---|---|
| juliasadler (57 harmful, 50 benign) | default | 36 (63.2%), 3 denied | 7 | 54 (94.7%), 3 denied | 8 |
| | strict | 41 (71.9%), 4 denied | 10 | 55 (96.5%), 4 denied | 11 |
| amandeavor (45 harmful, 45 benign) | default | 22 (48.9%), 17 denied | 0 | 32 (71.1%), 18 denied | 0 |
| | strict | 25 (55.6%), 19 denied | 2 | 35 (77.8%), 19 denied | 2 |
| both (102 harmful, 95 benign) | default | 58 (56.9%), 20 denied | 7 (7.4%) | 86 (84.3%), 21 denied | 8 (8.4%) |
| | strict | 66 (64.7%), 23 denied | 12 (12.6%) | 90 (88.2%), 23 denied | 13 (13.7%) |
| amandeavor (45 harmful, 45 benign) | default | 22 (48.9%), 17 denied | 0 | 36 (80.0%), 18 denied | 0 |
| | strict | 25 (55.6%), 19 denied | 2 | 39 (86.7%), 19 denied | 2 |
| both (102 harmful, 95 benign) | default | 58 (56.9%), 20 denied | 7 (7.4%) | 90 (88.2%), 21 denied | 8 (8.4%) |
| | strict | 66 (64.7%), 23 denied | 12 (12.6%) | 94 (92.2%), 23 denied | 13 (13.7%) |

The current policy scores above were refreshed on macOS. The latency rows below keep the recorded macOS baseline, and the
existing gates are unchanged.
Expand All @@ -558,7 +558,8 @@ scored: the one policy change between the two (#383) changes none of its numbers
call caught since is held by a rule written after its pool was published: for the first pool, the database-client
rules `TK-A039` to `TK-A041` (10 calls, #359), repository deletion `TK-A042` (4, #360), infrastructure removal in
`TK-A001` (3, #361) and a secret read through a cloud CLI, `TK-A037` (1, #372); for the second, 6 by `TK-A047`
(repository secrets and branch protection through `gh`, #402), 2 by `TK-C010` (turning off AWS audit logging, #403), and 2 CI deletions by `TK-A049` (#405). One benign call of the first pool is newly held on both
(repository secrets and branch protection through `gh`, #402), 2 by `TK-C010` (turning off AWS audit logging, #403), 2 CI deletions by `TK-A049` (#405) and 4 by `TK-A052` and
`TK-A053` (a Kubernetes or Vault secret read, #404). One benign call of the first pool is newly held on both
profiles: `TK-A039` holds `psql … -c "BEGIN; DROP TABLE staging_import; ROLLBACK;"`, whose drop is rolled back.

| Latency (µs) | p50 | p99 | max | p99_ref |
Expand All @@ -585,24 +586,25 @@ and 250 (59.5%); the credential-store, container, package-source and history rul
`TK-A028`) to 236 (56.2%) and 266 (63.3%); the fixes of the 1.4 review (`TK-A029`, `TK-A030`, the shells the parser
can't read, package sources set by a variable) to 237 (56.4%) and 266, with no new false positive on the held-out or
session calls; `TK-A035` to `TK-A038` (a secret's value read through a cloud CLI or an MCP
tool) to 240 (57.1%) and 269 (64.0%), with none either. The database-client rules (`TK-A039`, `TK-A040`, `TK-A041`) raised that to 245 (58.3%) and 274 (65.2%), again with no new false positive on the held-out or session calls. In 1.5.1, `TK-A046` (a download piped into an interpreter given code), `TK-D002` reading the parsed command, and `TK-A010`'s login scripts and cron files raised it to 247 (58.8%) and 275 (65.5%), with none either. `TK-A048` (deleting Docker or Podman volumes) raised it to 248 (59.0%) and 276 (65.7%), with none either. `TK-C010`'s AWS audit-logging commands (#403) raised it to 250 (59.5%) and 278 (66.2%), with none either. `TK-A055` (framework commands that drop or reset a database) left both unchanged, with no new false positive: the pool has none of those calls. Each rise came from rules written while the pool's misses were known, so read them as scores on a known
tool) to 240 (57.1%) and 269 (64.0%), with none either. The database-client rules (`TK-A039`, `TK-A040`, `TK-A041`) raised that to 245 (58.3%) and 274 (65.2%), again with no new false positive on the held-out or session calls. In 1.5.1, `TK-A046` (a download piped into an interpreter given code), `TK-D002` reading the parsed command, and `TK-A010`'s login scripts and cron files raised it to 247 (58.8%) and 275 (65.5%), with none either. `TK-A048` (deleting Docker or Podman volumes) raised it to 248 (59.0%) and 276 (65.7%), with none either. `TK-C010`'s AWS audit-logging commands (#403) raised it to 250 (59.5%) and 278 (66.2%), with none either. `TK-A055` (framework commands that drop or reset a database) left both unchanged, with no new false positive: the pool has none of those calls. `TK-A052` and `TK-A053` (a Kubernetes or Vault secret read, #404) raised it to 254 (60.5%) and 285 (67.9%), with none either. Each rise came from rules written while the pool's misses were known, so read them as scores on a known
pool, not as blind recall. The gates were
recorded again with those rules, on purpose: strict's false positives on E17's benign calls rose from 4.1% to 4.9%
(29 of 711 to 36 of 738). The benign cases added since, with the secret-read, repository-deletion,
repository-protection, database-client and infrastructure-removal rules, are calls strict does not hold, so the count
stayed at 36 while the pool grew to 814 (4.4%). Of the eight the 1.5.2 rule sweep added, strict holds one
(`kubectl delete pod`, as `TK-F004`), and of the seven its review added, three (`kubectl delete -f`,
`py -m pip install` and a `curl -T` upload); the five Docker-volume cases (`TK-A048`), the 21 AWS audit-logging reads and changes that leave detection on
(`TK-C010`) and the six lookalikes of `TK-A055` strict does not hold either: 46 of 866 (5.3%).
(`TK-C010`), the six lookalikes of `TK-A055` and the fifteen Kubernetes and Vault listings and look-alikes (`TK-A052`,
`TK-A053`) strict does not hold either: 46 of 881 (5.2%).
All 46 are benign calls strict holds (six more than 1.5.2's 40: 1.5.3's strict holds every install, push and network command its limits row names): 32 as a push, publish, install, apply or `kubectl delete` (`TK-F004`), 13 as a
network command (`TK-F013`), and 1 draft email to a list address (`TK-M003`); `TK-F012` (any URL a call acts on) only flags in
strict, it never holds one by itself.

Shell rewrites (`eval/e21_robustness.py`): E17's harmful calls are decided as they are, and each of the 478 shell
Shell rewrites (`eval/e21_robustness.py`): E17's harmful calls are decided as they are, and each of the 500 shell
commands among them again rewritten in up to five equivalent ways (`cd . &&` or `true;` before it, Bash's `command`
builtin, `bash -c` and `env bash -c` around it; a form that could change what the command does is left out, such as
`bash -c` around a `cd` or an `export`). A call is contained when the server pack still denies or holds it: all 806
originals and all 2,297 rewrites are, and both engines decide each the same. The run gates against a recorded
`bash -c` around a `cd` or an `export`). A call is contained when the server pack still denies or holds it: all 828
originals and all 2,405 rewrites are, and both engines decide each the same. The run gates against a recorded
baseline, rule id by rule id, so a rule that held a form and no longer does fails it. Results:
`eval/results/e21_robustness.json`.

Expand Down
Loading
Loading