Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
9434982
Ask before workflow commands upload secrets
kausthubhk Nov 7, 2026
5b6f040
Merge upstream main and address CI workflow review
kausthubhk Nov 10, 2026
21e1f36
Handle CI credential upload cases
kausthubhk Nov 11, 2026
204b17e
Refresh policy evaluation evidence
kausthubhk Nov 11, 2026
e9c86fd
Document CI egress rule limits
kausthubhk Nov 11, 2026
e1332c7
Address CI credential upload review findings
kausthubhk Nov 12, 2026
6c13a98
Merge upstream main into workflow upload policy
kausthubhk Nov 12, 2026
f564093
Correct workflow upload policy docs and refresh robustness results
kausthubhk Nov 12, 2026
68f5493
Check CI upload performance on both regex backends
kausthubhk Nov 12, 2026
85aa2d3
Merge main into agent/issue-405-workflow
LoopGlitch26 Nov 12, 2026
602c2b8
Policy: raw Docker or Podman socket calls and cgroup release_agent wr…
munjikesh Nov 11, 2026
7d005cf
Policy: TK-A064 reads time, !, env by path and option words before DO…
munjikesh Nov 12, 2026
5cd15e2
Speed test: an unspaced line of ;DOCKER_HOST=unix:// decides before t…
LoopGlitch26 Nov 12, 2026
dca9814
Coding pack 2026.11-v2-84; E17 and E21 re-recorded on main
LoopGlitch26 Nov 12, 2026
8377079
Coding pack 2026.11-v2-85; E17 and E21 re-recorded on main
LoopGlitch26 Nov 12, 2026
d9c1a55
Merge agent/carry-pr731 (#731's rules) into agent/carry-pr663
LoopGlitch26 Nov 12, 2026
b30928a
E17 and E21 re-recorded with #731's and this PR's rules together
LoopGlitch26 Nov 12, 2026
d6cb17a
TK-A061: read JSON bodies, more secret forms, CI long forms, command …
LoopGlitch26 Nov 13, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ your customers and auditors can verify the record themselves, offline.**
<table>
<tr>
<td align="center" width="25%"><h3>8 of 8</h3>tamper attacks caught<br><sub><a href="#proof">480 trials</a>, with a witness (E1)</sub></td>
<td align="center" width="25%"><h3>840 of 840</h3>harmful calls stopped<br><sub><a href="#proof">11 classes</a>, 1 false alarm in 890 (E17)</sub></td>
<td align="center" width="25%"><h3>859 of 859</h3>harmful calls stopped<br><sub><a href="#proof">11 classes</a>, 1 false alarm in 915 (E17)</sub></td>
<td align="center" width="25%"><h3>&lt; 1 ms</h3>added per tool call<br><sub><a href="#proof">0.91 ms at p99</a>, signer round trip (E9)</sub></td>
<td align="center" width="25%"><h3>8 of 8</h3>insider attacks caught<br><sub><a href="#proof">a merge gate</a> on every PR (E8)</sub></td>
</tr>
Expand Down Expand Up @@ -344,8 +344,8 @@ Our own experiments: small, synthetic and written by us. Nothing rounded up, and
| **Cross-check** (E10): a skipped hook, a made-up call, changed arguments | **12 of 12 flagged**, 0 false positives |
| **Three verifiers, one verdict** (E22): 63 bundles, 33 of them forged, each checked by the Python CLI, the TypeScript CLI and the browser page | **All three agree on all 63**; no forgery verifies. A 50,000-call run verifies in the browser in 14 s |
| **Overhead** (E9): signer round trip over a Unix socket | **0.45 ms p50, 0.91 ms p99** per call; 5,356 events/s |
| **Policy packs, known cases** (E17): 840 harmful, 890 benign calls | 840 stopped, 1 false alarm |
| **Policy packs, written blind to the rules** (E21): 420 harmful, 432 benign | 211 of 420 stopped or held on the one blind run (255 now, with rules added after reading the pool); 18 benign held |
| **Policy packs, known cases** (E17): 859 harmful, 915 benign calls | 859 stopped, 1 false alarm |
| **Policy packs, written blind to the rules** (E21): 420 harmful, 432 benign | 211 of 420 stopped or held on the one blind run (259 now, with rules added after reading the pool); 18 benign held |

That last row is on purpose. **Policy rules are tripwires, not a wall.** They match text, and a determined agent can
reword a command. What makes the record trustworthy is the signer's isolation and the witness, and those are what the
Expand Down
26 changes: 13 additions & 13 deletions docs/evaluation.md
Original file line number Diff line number Diff line change
Expand Up @@ -281,8 +281,8 @@ p99. `make eval` runs the `--quick` form (5 calls per phase instead of 20). Resu

Three offline parts, each with a signer in process; nothing is executed.

- **Policy packs** (`eval/e17_policy_corpus.py`): 840 harmful and 890 benign tool calls in 11 classes
(`tests/data/policy/`), written by the project, decided by `server.yaml` with both engines. All 840 harmful calls are denied or held; one benign call is held (a draft email to a list address, `TK-M003`), 1 in 890 (0.11%); the engines
- **Policy packs** (`eval/e17_policy_corpus.py`): 859 harmful and 915 benign tool calls in 11 classes
(`tests/data/policy/`), written by the project, decided by `server.yaml` with both engines. All 859 harmful calls are denied or held; one benign call is held (a draft email to a list address, `TK-M003`), 1 in 915 (0.11%); the engines
decide every call the same. It gates on every harmful call caught, at most 2% false positives and engine agreement.
- **Detection** (`eval/e17_detection.py`): 10 scenarios for [sequences](incidents.md): a sequence split across two
runs, a tool alias the policy doesn't map, parallel runs, a GET with a query string as egress, a denied step that
Expand Down Expand Up @@ -528,12 +528,12 @@ The recorded results, measured after those rule changes on a pool that is no lon

| Profile | Held-out caught | of which denied | caught only as an unmapped tool | Held-out benign FP | Session FP |
|---|---|---|---|---|---|
| default | 255 / 420 (60.7%) | 116 (27.6%) | 33 | 18 / 432 (4.2%) | 2 / 130 (1.5%) |
| strict | 285 / 420 (67.9%) | 154 (36.7%) | 33 | 56 / 432 (13.0%) | 20 / 130 (15.4%) |
| default | 259 / 420 (61.7%) | 116 (27.6%) | 33 | 18 / 432 (4.2%) | 2 / 130 (1.5%) |
| strict | 286 / 420 (68.1%) | 154 (36.7%) | 33 | 56 / 432 (13.0%) | 20 / 130 (15.4%) |

Weakest categories (default, caught): exfiltration 12 of 39, privilege escalation 8 of 27, prompt injection 8 of 22,
destructive 16 of 36, defence evasion 15 of 30, supply chain 16 of 27; strongest: database 32 of
32, Kubernetes 25 of 27. By evasion, calls with none are caught 155 of 275; quoting 9 of 18, environment prefixes 8 of 14, substitution 40
32, Kubernetes 25 of 27. By evasion, calls with none are caught 158 of 275; quoting 9 of 18, environment prefixes 9 of 14, substitution 40
of 49, interpreter one-liners 23 of 28 (many held as unreadable, `TK-SHELL-PARSE`; a program a variable names is read,
and one that is a bare program name, `K=kubectl; $K …`, is still held: it only hides that name). Per category, severity and evasion, and every miss
and false positive: `eval/results/e21_policy_robustness.json`. The two engines decided every call the same.
Expand All @@ -545,12 +545,12 @@ denied or held:
|---|---|---|---|---|---|
| juliasadler (57 harmful, 50 benign) | default | 36 (63.2%), 3 denied | 7 | 55 (96.5%), 3 denied | 8 |
| | strict | 41 (71.9%), 4 denied | 10 | 56 (98.2%), 4 denied | 11 |
| amandeavor (45 harmful, 45 benign) | default | 22 (48.9%), 17 denied | 0 | 36 (80.0%), 18 denied | 0 |
| | strict | 25 (55.6%), 19 denied | 2 | 39 (86.7%), 19 denied | 2 |
| amandeavor (45 harmful, 45 benign) | default | 22 (48.9%), 17 denied | 0 | 37 (82.2%), 18 denied | 0 |
| | strict | 25 (55.6%), 19 denied | 2 | 40 (88.9%), 19 denied | 2 |
| email_ops (25 harmful, 25 benign) | default | 5 (20.0%), 3 denied | 2 | 5 (20.0%), 3 denied | 2 |
| | strict | 6 (24.0%), 4 denied | 4 | 6 (24.0%), 4 denied | 4 |
| all three (127 harmful, 120 benign) | default | 63 (49.6%), 23 denied | 9 (7.5%) | 96 (75.6%), 24 denied | 10 (8.3%) |
| | strict | 72 (56.7%), 27 denied | 16 (13.3%) | 101 (79.5%), 27 denied | 17 (14.2%) |
| all three (127 harmful, 120 benign) | default | 63 (49.6%), 23 denied | 9 (7.5%) | 97 (76.4%), 24 denied | 10 (8.3%) |
| | strict | 72 (56.7%), 27 denied | 16 (13.3%) | 102 (80.3%), 27 denied | 17 (14.2%) |

The current policy scores above and the latency rows below were refreshed on macOS; the existing gates are unchanged.

Expand Down Expand Up @@ -598,15 +598,15 @@ stayed at 36 while the pool grew to 814 (4.4%). Of the eight the 1.5.2 rule swee
(`kubectl delete pod`, as `TK-F004`), and of the seven its review added, three (`kubectl delete -f`,
`py -m pip install` and a `curl -T` upload); the five Docker-volume cases (`TK-A048`), the 21 AWS audit-logging reads and changes that leave detection on
(`TK-C010`), the six lookalikes of `TK-A055` and the fifteen Kubernetes and Vault listings and look-alikes (`TK-A052`,
`TK-A053`) strict does not hold either: 46 of 890 (5.2%).
All 46 are benign calls strict holds (six more than 1.5.2's 40: 1.5.3's strict holds every install, push and network command its limits row names): 32 as a push, publish, install, apply or `kubectl delete` (`TK-F004`), 13 as a
`TK-A053`) strict does not hold either: 47 of 915 (5.1%).
All 47 are benign calls strict holds (seven more than 1.5.2's 40: 1.5.3's strict holds every install, push and network command its limits row names): 33 as a push, publish, install, apply or `kubectl delete` (`TK-F004`), 13 as a
network command (`TK-F013`), and 1 draft email to a list address (`TK-M003`); `TK-F012` (any URL a call acts on) only flags in
strict, it never holds one by itself.

Shell rewrites (`eval/e21_robustness.py`): E17's harmful calls are decided as they are, and each of the 511 shell
Shell rewrites (`eval/e21_robustness.py`): E17's harmful calls are decided as they are, and each of the 523 shell
commands among them again rewritten in up to five equivalent ways (`cd . &&` or `true;` before it, Bash's `command`
builtin, `bash -c` and `env bash -c` around it; a form that could change what the command does is left out, such as
`bash -c` around a `cd` or an `export`). A call is contained when the server pack still denies or holds it: all 840 originals and all 2,451 rewrites are, and both engines decide each the same. The run gates against a recorded
`bash -c` around a `cd` or an `export`). A call is contained when the server pack still denies or holds it: all 859 originals and all 2,509 rewrites are, and both engines decide each the same. The run gates against a recorded
baseline, rule id by rule id, so a rule that held a form and no longer does fails it. Results:
`eval/results/e21_robustness.json`.

Expand Down
Loading
Loading