Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ observation), with Python and TypeScript SDKs and framework integrations. [HIGH
| `tracekit/report.py` | `tracekit report`: a window of a signer's store on one page, built on the viewer's reader, check and review (experimental) | [HIGH CONFIDENCE] |
| `tracekit/why/` | `tracekit why`: causal investigation and replay | [HIGH CONFIDENCE] |
| `tracekit/pcm/` | PCM 0.1, proof-carrying messages (experimental): `__init__.py` (message, schema and grant checks, replay, `verify`), `claims.py` (claim evidence), `cli.py` (`tracekit pcm verify`, `tracekit pcm demo`), `tools.json` (the frozen tool placement table, part of the protocol); its signer RPCs (`pcm_*`, contract version 20) are in `tracekit/signer/` | [HIGH CONFIDENCE] |
| `tracekit/grants.py` | Intent grants (experimental, docs/intent-grants.md): the `payment.send` intent, amounts, the payment reader and the caveat and limit checks the signer (`grant_*` RPCs, contract version 22; releases in `_decide`), the CLI (`tracekit grant`, `approvals approve --grant`) and the verifier (`intent grants`) share; a grant is a PCM grant as it is | [HIGH CONFIDENCE] |
| `tracekit/findings_v2.py`, `tracekit/findings_text.py`, `tracekit/findings_effects.py`, `tracekit/parsers.py`, `tracekit/review_model.py` | `tracekit analyze --v2`: intent, reasoning and action checks (TK-X200..X204, X208) over a v2 run and its content vault, the text checks (TK-X205 and v1's say-vs-do checks) and the effect checks (TK-X206, TK-X207: kernel sensor effects joined to calls); `parsers.said()` reads what a model said before each tool use; `--review`: the reviewer model (TK-X210, warn only) | [HIGH CONFIDENCE] |
| `tracekit/storage/`, `tracekit/identity/`, `tracekit/transport/` | v2 storage (file, Postgres), caller identities (uid, token, mTLS, k8s service account, OIDC, WebAuthn), RPC transports | [HIGH CONFIDENCE] |
| `tracekit/daemon.py` | v1 signer daemon (tracekitd): socket, peer identity, run state, approvals, checkpoints | [HIGH CONFIDENCE] |
Expand Down
3 changes: 2 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ help:
@echo "make release-dry-run build every release artifact, SBOM and sum locally, publish nothing, and check them (needs the package index and npm)"
@echo "make demo run the scripted end-to-end demo"
@echo "make demos record the offline demos in docs/demos/ again (scripts/record_demos.py; --check checks them)"
@echo "make eval offline evaluations E1 (integrity), E2 (overhead), E3 (policy gate), E4 (seeded faults, v1 and v2), E6 (findings), E17 (server policy corpora, detection), E18 (agent approvers), E19 (egress observation), E20 (intent, reasoning and action checks), E21 (policy robustness: held-out calls, and shell rewrites), on Linux E9 --quick (signer perf); rewrites eval/results/"
@echo "make eval offline evaluations E1 (integrity), E2 (overhead), E3 (policy gate), E4 (seeded faults, v1 and v2), E6 (findings), E17 (server policy corpora, detection), E18 (agent approvers), E19 (egress observation), E20 (intent, reasoning and action checks), E21 (policy robustness: held-out calls, and shell rewrites), E23 (intent grants), on Linux E9 --quick (signer perf); rewrites eval/results/"
@echo "make eval-agents E5: real Claude Code runs (needs the claude CLI; spends model usage)"
@echo "make eval-scale E7: SQL index over a million-event ledger (contrib/query)"
@echo "E8 (insider attacks) needs root and a Linux system-mode signer; CI runs it, see eval/e8_insider.py"
Expand Down Expand Up @@ -82,6 +82,7 @@ eval:
$(PY) eval/e20_alignment.py
$(PY) eval/e21_policy_robustness.py
$(PY) eval/e21_robustness.py
$(PY) eval/e23_grants.py
if [ "$$(uname)" = Linux ]; then $(PY) eval/e9_signer_perf.py --quick; fi

eval-agents:
Expand Down
7 changes: 7 additions & 0 deletions docs/approvals.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,13 @@ unchanged: the hook only chooses which arguments the retry is decided with, and
the approval. So a retry exactly as approved, description included, runs under that approval even when another
wording of the command was rejected: the person approved that call. It runs once; the approval is used up then.

## Approve calls like this one: grants

For a held payment, `tracekit approvals approve ID --grant --payee … --max '200 USDC' --total '500 USDC' --for 1h`
approves the call and grants the run more payments like it, within those limits: they run without a stop, and
anything outside them is held with the reason. `tracekit approvals show ID` says how a grant reads the call. See
[intent grants](intent-grants.md).

## Tiers: T1 and T2 executors

- **T1** (default): the approval binds the arguments the agent reported. The adapter runs the call in the agent's
Expand Down
18 changes: 16 additions & 2 deletions docs/evaluation.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
# Evaluation

Twenty-one experiments, E1 to E22 (there is no E13). `make eval` runs the offline ones: E1 to E4 (E4 on v1 and v2
bundles), E6, E10, E11, E12, E14 (as `--quick`; POSIX), E16, E17, E18, E19, E20 and E21, and E9 `--quick` on Linux.
Twenty-two experiments, E1 to E23 (there is no E13). `make eval` runs the offline ones: E1 to E4 (E4 on v1 and v2
bundles), E6, E10, E11, E12, E14 (as `--quick`; POSIX), E16, E17, E18, E19, E20, E21 and E23, and E9 `--quick` on
Linux.
E5 uses real Claude Code runs and is opt-in (`make eval-agents`, needs the `claude` CLI and spends model usage); E7 is
`make eval-scale`; E22 (`eval/e22_verifier_agreement.py`) runs on its own. In CI, E8 (insider attacks, which needs root
and a Linux system-mode signer) is a merge gate, and E9 (`eval/e9_signer_perf.py`, latency and throughput over the Unix
Expand Down Expand Up @@ -645,6 +646,19 @@ rewrite let a log origin widen it past a phone's screen. Not in `make eval`: it
Chrome and each framework (`--python NAME=PY` runs an example with an interpreter that has it; an example that can't
run is listed as skipped). Firefox and Safari are not run.

## E23: intent grants

`eval/e23_grants.py` runs a payout agent against one in-process signer whose payment rule opts into
[intent grants](intent-grants.md), with a named approver. Seven parts: 30 payments, 24 within the grant and 6 outside it
(an unknown payee, over the most for one payment, past the total), where the person's stops fall from 30 to 7 and
nothing outside the limits runs; 25 calls a grant must not read as within it (amounts in other units, hundredths of a
USDC past the cap, a currency with no known unit, payee look-alikes, a second payee, arguments that can move money or
pick what is paid, such as a fee, a split, a memo, an invoice or order id, another currency, another tool), all held;
the agent issuing a grant
for its own run, passing one on, or asking for one past a day, all refused; 100 payments of 7 USDC under a 50 USDC
total, of which 7 run; a revoked and an expired grant, which release nothing; totals across a signer restart; and the
run's bundle, whose `intent grants` line re-checks every release. Results: `eval/results/e23_grants.json`.

## What is not measured

Signer throughput with a separate-user signer or over the remote gateway, durability across crashes beyond the
Expand Down
19 changes: 17 additions & 2 deletions docs/format-v2.md
Original file line number Diff line number Diff line change
Expand Up @@ -575,7 +575,7 @@ them, all critical; from 1.4.0 the bundle layouts it reads, critical too (`layou
| 1.2.0 | `run.child`, `approval.escalated`, `approval.revoked`, `approval.suggested`, `approval.agents`, `incident.opened`, `incident.updated`, `incident.capped`, `run.held`, `run.resumed`, `run.quarantined`, `incident.resolved`, `incident.response`, `identity.suspension`, `effect.observed` | `run.registered`: `parent_run`, `session`, `posture`; `capture.gap`: `run_id`; `run.final`: `provider_executed`, `paired_by_args`, `matrix`; `policy.decision`: `via`, `caller`, `detect`, `incident_id`; `approval.request`: `agent`, `via`, `incident_id`, `passkey`; `tool.result`, `input.observed`: `via`; `approval`: `approver_kind`, `asserted`, `spent`, `not_before`, `consume_by`, `incident_id`; `approval.expired`: `reason`; `reconcile.unexplained_effect`: `observer`, `kind`, `host`, `port` | `capture.gap` `kind`: `decided_not_completed`, `late_activity`, `concurrent_run`, `tailer_redelegated`, `reask_after_reject`, `held_call_unresolved`, `observer_lost`, `effect_outside_run`; `run.closing` `reason`: `quarantine` | — |
| 1.3.0 | `reconcile.result_mismatch` | — | `capture.gap` `kind`: `result_unchecked` | — |
| 1.4.0 | `constraint.added` | `effect.observed`: `sensor`, `denied`; `review`: `cites`; `model.exchange`: `said_only`; `capture.gap`: `sensor` | `capture.gap` `kind`: `constraint_refused` | `amended`, `openshell.policy`; and the slot itself (`ext` fields, `ext` records: critical) |
| 1.6.0 | — | — | — | `box`: on `run.registered`, `{id, image_digest, allow, engine, channel?}`, the label of a Tracekit Box box's run (a `box_daemons` uid's) and, copied with `parent_run`, of each run its agent channel registered; the signer takes the engine from its config. Read as `boxed agent`: `PROOF AGAINST THE BOXED AGENT (micro-VM)` and no more |
| 1.6.0 | — | — | — | `box`: on `run.registered`, `{id, image_digest, allow, engine, channel?}`, the label of a Tracekit Box box's run (a `box_daemons` uid's) and, copied with `parent_run`, of each run its agent channel registered; the signer takes the engine from its config. Read as `boxed agent`: `PROOF AGAINST THE BOXED AGENT (micro-VM)` and no more; `intent.grant` and `intent.revoked` (`ext` records of source `signer`) and `grant` (on a `policy.decision`), a person's [intent grant](intent-grants.md) and the payments it released |

Layouts (critical): 1.4.0 adds `run parts`, a run in parts (above).

Expand Down Expand Up @@ -613,6 +613,10 @@ a new critical row, so a verifier from 1.4.0 on keeps verifying later bundles:
- `box` is non-critical: a box channel's run records `signer_isolation: unknown`, so a verifier that doesn't read the
label claims nothing for it; one that reads it adds a `boxed agent` line, which narrows what the run proves (against
the boxed agent only) and never widens it.
- `intent.grant`, `intent.revoked` and `grant` are non-critical: a payment a grant released is a `flag` decision
either way, so a verifier that doesn't read them says the call ran without a stop, which is true; one that reads them
adds an `intent grants` line that re-checks each release against its grant ([intent grants](intent-grants.md)), and
fails the verification when the signer released a payment its own grant holds.
- `review.cites` and `effect.observed.sensor` stay critical: a verifier that passed over a signed finding's citations
would label it `imported`, a false line; and a sensor's effects are records of what the agent's own processes did,
in its run, which a verifier must read to verify that run. So does `capture.gap.sensor`: a 1.3 verifier counts every
Expand Down Expand Up @@ -733,7 +737,18 @@ line and verdict is in [verdicts](verdicts.md).
policy`, passing with `OpenShell changed the sandbox policy or configuration n time(s) (<change>: <count>, ... the
first 8 in code-point order), as OpenShell reported it`, a warning listing the records (the first 20) that carry it
other than as an `ext` record of source `signer` with a `change` of `[a-z0-9_.-]{1,64}` and a non-empty
`observer`. `box` (1.6.0): `boxed agent`, a warning listing the records (the first 20) when any record carries it
`observer`. `intent.grant`, `intent.revoked` and `grant` (1.6.0), read together with every `policy.decision`
whose `rule_ids` name `TK-GRANT`: `intent grants`, failing, listing the records (the first 20), where a released
payment (a `flag` with `TK-GRANT`) names no grant the bundle holds earlier in its run, or one whose record fails
the grant rules below, or breaks its grant: its fields aren't whole numbers of the intent's, its tool isn't one
the grant covers, its grant was revoked or expired before it or its fields break a caveat or a limit, its totals
aren't the previous plus its own as whole numbers, or its tier isn't the record's; and where a decision names
`TK-GRANT` without `ext.grant`. A warning, listing the records, where a grant isn't a person's PCM grant for its own
run and tenant, never passed on, its `id` its hash's, naming 1 to 16 tools, valid for its intent in a currency with
a known unit with an expiry within 24 h of its `ts`, and its first totals those of no call or of the held call it
came from; a revocation names no grant live earlier in its run; or an `ext.grant` isn't on a `flag` with
`TK-GRANT`. Else it passes with the grants, who issued them, and the payments released at T2 and T1.
`box` (1.6.0): `boxed agent`, a warning listing the records (the first 20) when any record carries it
other than on a `run.registered` whose `data.identity.scheme` is `uid`, as an object with exactly `id`
(`[A-Za-z0-9._:-]{1,128}`), `image_digest` (`sha256:` and 64 hex digits), `allow` (a list of strings), `engine`
(`firecracker`, read as `micro-VM`) and optionally `channel` (`agent`). Else only the labels on the first record of
Expand Down
Loading
Loading