Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ and gives you one `.tkb` file per run that anyone can verify offline.**
<table>
<tr>
<td align="center" width="25%"><h3>8 of 8</h3>tamper attacks caught<br><sub><a href="#proof">480 trials</a>, with a witness (E1)</sub></td>
<td align="center" width="25%"><h3>699 of 699</h3>harmful calls stopped<br><sub><a href="#proof">11 classes</a>, 1 false alarm in 800 (E17)</sub></td>
<td align="center" width="25%"><h3>715 of 715</h3>harmful calls stopped<br><sub><a href="#proof">11 classes</a>, 1 false alarm in 814 (E17)</sub></td>
<td align="center" width="25%"><h3>&lt; 1 ms</h3>added per tool call<br><sub><a href="#proof">0.91 ms at p99</a>, signer round trip (E9)</sub></td>
<td align="center" width="25%"><h3>8 of 8</h3>insider attacks caught<br><sub><a href="#proof">a merge gate</a> on every PR (E8)</sub></td>
</tr>
Expand Down Expand Up @@ -212,7 +212,7 @@ is tested, but may change between releases ([what's stable](https://github.com/C
| Core | What it does | Measured |
|---|---|---|
| **Record** | Every tool call, decision and result in one signed hash chain per run. A lost event becomes a signed gap, never silence | 6 of 8 tamper attacks caught by the log alone (E1) |
| **Decide** | Allow, deny, or hold each call for a person, before it runs. [Packs](https://github.com/Cygnux-Labs/Tracekit/wiki/Policy) for shell, files, SQL, HTTP, cloud CLIs, browsers, email, messaging, payments, secrets and MCP | 699 of 699 harmful calls stopped, 1 false alarm in 800 (E17) |
| **Decide** | Allow, deny, or hold each call for a person, before it runs. [Packs](https://github.com/Cygnux-Labs/Tracekit/wiki/Policy) for shell, files, SQL, HTTP, cloud CLIs, browsers, email, messaging, payments, secrets and MCP | 715 of 715 harmful calls stopped, 1 false alarm in 814 (E17) |
| **Approve** | A held call runs only after a yes from the command line, bound to its exact arguments and used once ([approvals](https://github.com/Cygnux-Labs/Tracekit/wiki/Approvals)) | Edited arguments, a replayed snapshot or another run's approval: refused, in the contract tests every stable framework integration runs |
| **Prove** | `tracekit verify` checks a run's file offline against keys you pinned | 8 of 8 insider attacks caught (E8) |
| **Watch** | `tracekit view`: every run verified, blocked and held calls, approvals, gaps, replay and a run review ([viewer](https://github.com/Cygnux-Labs/Tracekit/wiki/Viewer)) | Not measured |
Expand Down Expand Up @@ -240,7 +240,7 @@ Our own experiments: small, synthetic and written by us. Nothing rounded up. The
|---|---|---|
| **E1** tamper detection | 120-record signed logs, 8 attacks, 60 trials each, re-run on 1.2 | 6 of 8 caught by the log alone, **8 of 8 with a witness** |
| **E8** insider attacks | Real unprivileged users against a Linux system-mode signer: a decoy signer, a stopped signer, a swapped policy, fabricated runs | **8 of 8 caught**, a merge gate on every pull request |
| **E17** policy packs | 699 harmful and 800 benign tool calls in 11 classes, decided by both regex engines | **699 of 699 stopped**; 1 false alarm, a draft email held for a person |
| **E17** policy packs | 715 harmful and 814 benign tool calls in 11 classes, decided by both regex engines | **715 of 715 stopped**; 1 false alarm, a draft email held for a person |
| **E21** policy robustness | 420 harmful and 432 benign calls written blind to the rules; 130 developer-session calls | server pack: 211 of 420 stopped or held on the one blind run, 247 after rules were added that target the pool (read since); 18 of 432 false alarms, 2 of 130 in sessions; p99 185 µs |
| **E10** reconciliation | 30 runs, 12 with a seeded discrepancy: a missing hook, a fabricated call, changed arguments | **12 of 12 flagged**, 0 false positives |
| **E9** overhead | Signer round trip over a Unix socket, file storage, macOS quick run | **0.45 ms p50, 0.91 ms p99** per call; 5,356 events/s |
Expand Down
2 changes: 1 addition & 1 deletion docs/concepts/policy.md
Original file line number Diff line number Diff line change
Expand Up @@ -167,7 +167,7 @@ tracekit policy backtest --policy policy.yaml --store DATA_DIR/store # replay

Rules are tripwires, not a sandbox. A determined agent can phrase a command so no rule matches: the
[evaluation](../evaluation.md#e3-policy-gate) lists misses such as `X=rm; $X -rf ~` and a base64 payload piped to a
shell. On the shipped packs' own test corpus (699 harmful and 800 benign calls across 11 classes, written by the
shell. On the shipped packs' own test corpus (715 harmful and 814 benign calls across 11 classes, written by the
project) every harmful call is denied or held, with one false alarm. On E21's held-out pool (420 harmful and 432
benign calls, written blind to the rules), the only blind run caught 211 harmful calls (50.2%) with the server pack and
246 (58.6%) with the strict pack. With rules added since, written while the misses were known, they catch 245 (58.3%)
Expand Down
2 changes: 1 addition & 1 deletion docs/concepts/security-model.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ synthetic and written by the project.
| Reconciliation (E10) | 12 of 12 seeded discrepancies flagged, 0 false positives |
| Saved-state tampering (E11) | 21 of 21 caught |
| Tenant isolation (E12) | Every attack refused, no leaks under concurrency |
| Policy packs on the project's corpus (E17) | 699 of 699 harmful calls stopped, 1 false alarm in 800 |
| Policy packs on the project's corpus (E17) | 715 of 715 harmful calls stopped, 1 false alarm in 814 |
| Policy packs on calls written blind to the rules (E21) | Server pack: 211 of 420 harmful calls stopped or held on the only blind run; 245 with rules added since (the pool is no longer blind); 18 false alarms in 432 |
| Agent approvers (E18) | 7 of 7 attacks caught |
| Unrequested edits behind cover text, checked by `tracekit analyze --v2` (E20, synthetic; the blind pool v3, scored once) | Caught under every cover; after a "don't touch X", 79 to 100% by cover (67 to 92% held before the edit ran); 5 false findings and 1 held call in 1,812 honest calls |
Expand Down
16 changes: 8 additions & 8 deletions docs/evaluation.md
Original file line number Diff line number Diff line change
Expand Up @@ -279,9 +279,9 @@ p99. `make eval` runs the `--quick` form (5 calls per phase instead of 20). Resu

Three offline parts, each with a signer in process; nothing is executed.

- **Policy packs** (`eval/e17_policy_corpus.py`): 699 harmful and 800 benign tool calls in 11 classes
(`tests/data/policy/`), written by the project, decided by `server.yaml` with both engines. All 699 harmful calls are
denied or held; one benign call is held (a draft email to a list address, `TK-M003`), 1 in 800 (0.12%); the engines
- **Policy packs** (`eval/e17_policy_corpus.py`): 715 harmful and 814 benign tool calls in 11 classes
(`tests/data/policy/`), written by the project, decided by `server.yaml` with both engines. All 715 harmful calls are
denied or held; one benign call is held (a draft email to a list address, `TK-M003`), 1 in 814 (0.12%); the engines
decide every call the same. It gates on every harmful call caught, at most 2% false positives and engine agreement.
- **Detection** (`eval/e17_detection.py`): 10 scenarios for [sequences](incidents.md): a sequence split across two
runs, a tool alias the policy doesn't map, parallel runs, a GET with a query string as egress, a denied step that
Expand Down Expand Up @@ -553,17 +553,17 @@ tool) to 240 (57.1%) and 269 (64.0%), with none either. The database-client rule
pool, not as blind recall. The gates were
recorded again with those rules, on purpose: strict's false positives on E17's benign calls rose from 4.1% to 4.9%
(29 of 711 to 36 of 738). The benign cases added since, with the secret-read, repository-deletion,
database-client and infrastructure-removal rules, are calls strict does not hold, so the count stayed at 36 while the
pool grew to 800 (4.5%).
repository-protection, database-client and infrastructure-removal rules, are calls strict does not hold, so the count
stayed at 36 while the pool grew to 814 (4.4%).
All 36 are benign calls strict holds: 23 as a push, publish, install or apply (`TK-F004`), 12 as a network command
(`TK-F013`), and 1 draft email to a list address (`TK-M003`); `TK-F012` (any URL a call acts on) only flags in
strict, it never holds one by itself.

Shell rewrites (`eval/e21_robustness.py`): E17's 699 harmful calls are decided as they are, and each of the 373 shell
Shell rewrites (`eval/e21_robustness.py`): E17's 715 harmful calls are decided as they are, and each of the 389 shell
commands among them again rewritten in up to five equivalent ways (`cd . &&` or `true;` before it, Bash's `command`
builtin, `bash -c` and `env bash -c` around it; a form that could change what the command does is left out, such as
`bash -c` around a `cd` or an `export`). A call is contained when the server pack still denies or holds it: all 699
originals and all 1,792 rewrites are, and both engines decide each the same. The run gates against a recorded
`bash -c` around a `cd` or an `export`). A call is contained when the server pack still denies or holds it: all 715
originals and all 1,871 rewrites are, and both engines decide each the same. The run gates against a recorded
baseline, rule id by rule id, so a rule that held a form and no longer does fails it. Results:
`eval/results/e21_robustness.json`.

Expand Down
Loading
Loading