Skip to content
65 changes: 65 additions & 0 deletions docs/audit.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
# `tracekit audit`

`tracekit audit` is a read-only command that replays the tool calls found in local coding-agent transcripts through the policy engine and reports what the current policy would have held or denied.

It does not start a signer, does not write anything, and does not make network calls. It is a replay against today's policy, not a signed record. Tracekit never recorded these calls, so nothing here is evidence.

It is experimental ([support](support.md)): it works and is tested, but may change.

## Usage

```
tracekit audit [ROOT] [--policy PATH] [--since 7d] [--show] [--all] [--json]
```

```
$ tracekit audit --since 7d --show
a replay against today's policy, not a signed record
policy: …/tracekit/policy2/packs/dev.yaml (sha256:…)

1204 tool calls in 31 sessions since 7d: Tracekit would have denied 2 and held 9 for a person; it would have flagged 57 and allowed 1136.
…
by rule:
TK-F003: 41 (destructive)
TK-A002: 3 (deleting a remote Git ref)
TK-D004: 1 (recursive delete of root or home)
…
denied and held calls (--all for every call):
4f1c… 2026-10-28T14:02:11+00:00 Bash deny rules=TK-D004,TK-F003 text='Bash rm -rf ~/'
```

(The numbers above are an example, not a measurement.)

- `ROOT` defaults to `~/.claude/projects`; a folder that doesn't exist is an error (exit 2).
- `--policy` selects a policy file (default: the dev pack, `tracekit/policy2/packs/dev.yaml`).
- `--since` limits the replay (e.g. `7d`, `24h`, `30m`, or an ISO-8601 time, UTC unless it gives an offset); a call
without a timestamp is kept.
- `--show` includes each call as an approver sees it: its secrets redacted as the signer redacts them (`.env` content
too), cut to 72 characters.
- `--all` lists every call; without it, only the calls Tracekit would have denied or held for a person.
- `--json` emits a machine-readable report, with every call.

The report always begins with the disclaimer: "a replay against today's policy, not a signed record".

## What it reads

Every `.jsonl` file under ROOT, at any depth: each Claude Code session's transcript, and its subagents' files
(`<session>/subagents/*.jsonl`), which count under that session. Each line is a JSON object; assistant messages may contain `tool_use` blocks. The reader
reuses `tracekit.hook.transcript_entries` and the policy engine from `tracekit.policy2.engine`.

It follows no symlink. Files and folders it can't read are counted and skipped, and so is a line that isn't JSON.

## Output

- One line first: how many calls in how many sessions, and how many Tracekit would have denied, held, flagged and
allowed.
- Counts per verdict (`deny`, `ask` / held, `flag`, `allow`). A call the policy engine fails on counts as a deny
(`TK-ENGINE-ERROR`), as the signer would decide it.
- Counts per rule id, most first, each with what the rule is (its `reason` or `label`, or the viewer's words for the
engine's own rules).
- The session and timestamp of each denied or held call (every call with `--all`), and its redacted text with
`--show`.

## Limitations

It sees only the tool calls the transcript holds, decided by today's policy. A transcript the agent could edit is not evidence. See [limits](limits.md#audit-tracekit-audit).
9 changes: 9 additions & 0 deletions docs/limits.md
Original file line number Diff line number Diff line change
Expand Up @@ -773,6 +773,15 @@ What Tracekit guarantees and doesn't, per deployment, on one page: [guarantees](
| Before it places or counts a run, it checks every record of the store against the newest checkpoint, in every tenant (`--tenant` too) and the signer's own runs: its hash is its event's, it is filed under the run its event names, and the hashes make the checkpoint's root (3.6 seconds for a store of 246,344 records and 845 runs on a laptop, whatever `--since` or `--tenant`). A run holding an edited record is listed `FAILED` whatever its tenant and times, a signer's run or another tenant's included; when the hashes don't make the root (a hash was rewritten too), the store and every run fail; and the command exits 1. Records newer than the checkpoint are not checked against it and place no run: a run whose covered records all fall before the window isn't listed, even when newer records of it fall in the window. The report says how many records of the store are newer, and a run no checkpoint covers yet is `PENDING`. | The store files a record under the tenant, run id and time the record holds, and only the signed checkpoint vouches for them; in dev mode the agent's user can edit the store. | Run it again after the next checkpoint (the signer writes one within seconds); for a `FAILED` run, `tracekit view` shows its first failing check. | [viewer](viewer.md#weekly-report) |
| It reads the v2 signer's store only, not the v1 ledger (`tracekitd`), why logs or bundles. It summarises no signer's run (its key records, content reads, refused writes, a witness that didn't countersign): it checks their records and lists one only when it fails. | It summarises the agents' runs of the store `tracekit view` verifies. | `tracekit view` shows the signer's own rows; `tracekit view --ledger` the v1 ledger. | [viewer](viewer.md#laptop-viewer) |

## Audit (`tracekit audit`)

| Limit | Why it is so | What to do | Details |
|---|---|---|---|
| It reads Claude Code's transcripts only, and sees only the tool calls they hold: nothing the agent did outside a tool call, and nothing from another harness (Codex, Gemini CLI). | Each harness writes its transcripts in its own format; Claude Code's is the one it reads. | For what an agent does from now on, record it with Tracekit (`tracekit init --dev --v2`). | [audit](audit.md) |
| Its counts are evidence of nothing: the agent could have edited the transcript, and each call is decided by today's policy, not the one in force when it ran. | Tracekit never recorded these calls; only a signed record shows what happened. | Use it to see what Tracekit would hold; for evidence, record runs with the signer and export bundles. | [audit](audit.md) |
| It can count fewer holds than the signer would: each call is decided on its own, without the run's state, so `from: untrusted` rules never match (there is no provenance), and detect sequences, limits and the person's prompt constraints don't apply. | It reads a transcript, not a run; that state exists only in the signer. | Treat its counts as a floor; the signer's own decisions are the ones that hold. | [audit](audit.md) |
| It keeps every call it reads in memory (about 0.5 KB each; `--show` text is cut to 72 characters), so a very large transcript folder takes memory in proportion. | The JSON report and `--all` list every call, so it keeps them all until the counts are done. | Use `--since` to replay a shorter window. | [audit](audit.md) |

## Install

| Limit | Why it is so | What to do | Details |
Expand Down
1 change: 1 addition & 0 deletions docs/support.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ experimental.
| `tracekit why`: investigation, replay, signed findings | [why](why.md) |
| `tracekit graph`: graph and report built from v2 bundles | [graph](graph.md) |
| `tracekit report`: a week of a signer's store on one page, as text or JSON | [viewer](viewer.md#weekly-report) |
| `tracekit audit`: Claude Code transcripts replayed through the policy, read-only and not a record | [audit](audit.md) |
| Detection and containment: sequences and limits, holds, quarantine, suspensions, incidents, the webhook outbox | [incidents](incidents.md), [webhooks](webhooks.md) |
| Argument provenance (`untrusted`, `from_untrusted`) | [policy](policy-v2.md#provenance-untrusted-and-from-untrusted) |
| Readable reasoning: what the model said, recorded by the transcript tailer, the coding-agent hooks, the framework integrations and the LLM gateway | [reasoning](reasoning.md) |
Expand Down
4 changes: 2 additions & 2 deletions scripts/wiki.py
Original file line number Diff line number Diff line change
Expand Up @@ -41,8 +41,8 @@
"concepts/deployment.md", "concepts/storage-and-performance.md", "concepts/integrations.md",
"concepts/security-model.md", "architecture.md", "why-architecture.md"]),
("Integrations", ["integrations.md", "adapters.md", "otel.md", "remote-ingest.md", "integrations/*.md"]),
("Policy and approvals", ["policy-v2.md", "approvals.md", "pcm.md", "findings.md", "reasoning.md", "incidents.md",
"webhooks.md"]),
("Policy and approvals", ["policy-v2.md", "approvals.md", "pcm.md", "findings.md", "audit.md", "reasoning.md",
"incidents.md", "webhooks.md"]),
("Deploy and operate", ["deploy.md", "deploy-container.md", "deploy-kubernetes.md", "deploy-compose.md",
"identity.md", "executor.md", "a2a.md", "c2c.md", "mcp-gateway.md", "gateway.md", "egress.md", "witnesses.md",
"monitor.md", "issuer.md", "observability.md", "viewer.md", "horizon.md", "viewer-1.4.md",
Expand Down
153 changes: 153 additions & 0 deletions tests/test_audit.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,153 @@
"""Tests for `tracekit audit` (issue #490)."""
from __future__ import annotations

import contextlib
import io
import json
import os
import tempfile
import unittest
from datetime import datetime, timedelta, timezone
from pathlib import Path
from unittest import mock

from tracekit import audit, cli


def _call(tool, args, days_ago=0):
"""An assistant line with one tool use, as Claude Code writes it."""
at = datetime.now(timezone.utc) - timedelta(days=days_ago)
return {"type": "assistant", "timestamp": at.isoformat().replace("+00:00", "Z"),
"message": {"content": [{"type": "tool_use", "name": tool, "input": args}]}}


def _write_jsonl(path: Path, lines: list[dict | str]):
path.parent.mkdir(parents=True, exist_ok=True)
with path.open("w", encoding="utf-8", newline="\n") as f:
for line in lines:
f.write((line if isinstance(line, str) else json.dumps(line)) + "\n")


def _run(*argv):
out, err = io.StringIO(), io.StringIO()
with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err):
code = cli.main(["audit", *argv])
return code, out.getvalue(), err.getvalue()


class TestAudit(unittest.TestCase):
def setUp(self):
self.tmp = tempfile.TemporaryDirectory()
self.root = Path(self.tmp.name)

def tearDown(self):
self.tmp.cleanup()

def test_held_denied_allowed_and_malformed(self):
_write_jsonl(self.root / "proj" / "session-abc.jsonl", [
_call("Bash", {"command": "rm -rf /"}), _call("Bash", {"command": "terraform destroy -auto-approve"}),
_call("Bash", {"command": "ls -la"}), "this is not json", {"type": "user", "message": {"content": "hi"}}])
report = audit.audit(str(self.root))
self.assertEqual(report["counts"], {"deny": 1, "ask": 1, "flag": 0, "allow": 1})
self.assertIn("TK-D004", report["by_rule"])
self.assertIn("TK-A001", report["by_rule"])
self.assertEqual(report["skipped_malformed"], 1)
self.assertEqual({c["session"] for c in report["calls"]}, {"session-abc"})
self.assertTrue(all(c["time"] for c in report["calls"]))
self.assertNotIn("text", report["calls"][0]) # the command text only with --show
self.assertIn("not a signed record", report["disclaimer"])

def test_a_subagents_calls_count_under_its_session(self):
_write_jsonl(self.root / "proj" / "s1.jsonl", [_call("Bash", {"command": "ls"})])
_write_jsonl(self.root / "proj" / "s1" / "subagents" / "agent-a.jsonl", [_call("Bash", {"command": "ls"})])
report = audit.audit(str(self.root))
self.assertEqual((report["sessions"], {c["session"] for c in report["calls"]}), (1, {"s1"}))

def test_a_call_the_engine_fails_on_is_a_deny_as_the_signer_decides_it(self):
_write_jsonl(self.root / "s.jsonl", [_call("Bash", {"command": "ls"})])
with mock.patch("tracekit.policy2.engine.Engine.decide", side_effect=RuntimeError("boom")):
report = audit.audit(str(self.root))
self.assertEqual(report["counts"]["deny"], 1)
self.assertEqual(report["by_rule"], {"TK-ENGINE-ERROR": 1})
self.assertEqual(report["skipped_malformed"], 0)

def test_an_unreadable_file_is_counted_and_skipped(self):
bad = self.root / "proj" / "bad.jsonl"
_write_jsonl(bad, [_call("Bash", {"command": "rm -rf /"})])
bad.chmod(0)
if os.access(bad, os.R_OK): # root, or a file system that ignores the mode (Windows)
self.skipTest("the file stays readable here")
report = audit.audit(str(self.root))
self.assertEqual(len(report["unreadable"]), 1)
self.assertEqual(report["calls"], [])

def test_no_symlink_is_followed_but_a_root_reached_through_one_is_read(self):
with tempfile.TemporaryDirectory() as outside:
real = Path(outside) / "other.jsonl"
_write_jsonl(real, [_call("Bash", {"command": "rm -rf /"})])
try:
(self.root / "link.jsonl").symlink_to(real)
(self.root / "linkdir").symlink_to(outside, target_is_directory=True)
except OSError: # Windows without the privilege
self.skipTest("no symlinks here")
self.assertEqual(audit.audit(str(self.root))["calls"], [])
self.assertEqual(audit.audit(str(self.root / "linkdir"))["counts"]["deny"], 1) # a root that is a symlink

def test_since_keeps_only_newer_calls(self):
_write_jsonl(self.root / "s.jsonl", [_call("Bash", {"command": "echo old"}, days_ago=30),
_call("Bash", {"command": "echo new"}, days_ago=1)])
ten_days_ago = datetime.now(timezone.utc) - timedelta(days=10)
for since in ("10d", "240h", ten_days_ago.astimezone(timezone(timedelta(hours=5))).isoformat(),
ten_days_ago.replace(tzinfo=None).isoformat()): # an offset, and none (UTC)
report = audit.audit(str(self.root), since=since, show=True)
self.assertEqual([c["text"] for c in report["calls"]], ["Bash echo new"], since)
with self.assertRaises(ValueError):
audit.audit(str(self.root), since="last week")

def test_show_redacts_secrets_as_the_signer_does(self):
key = "sk-ant-" + "a1B2c3D4e5F6g7H8i9J0" * 2
_write_jsonl(self.root / "s.jsonl", [
_call("Bash", {"command": f"export ANTHROPIC_API_KEY={key}"}),
_call("Bash", {"command": "cat > .env <<'EOF'\nSTRIPE_WEBHOOK=whsec_abc123\nEOF"})])
key_text, env_text = (c["text"] for c in audit.audit(str(self.root), show=True)["calls"])
self.assertNotIn(key, key_text)
self.assertIn("export ANTHROPIC_API_KEY=", key_text)
self.assertNotIn("whsec_abc123", env_text) # a .env write: every value, as the signer redacts it
self.assertIn("STRIPE_WEBHOOK=", env_text)

def test_the_command_prints_the_report_and_json_matches_it(self):
_write_jsonl(self.root / "s.jsonl", [_call("Bash", {"command": "rm -rf /"}), _call("Bash", {"command": "ls"})])
report = audit.audit(str(self.root))
code, out, err = _run(str(self.root), "--json")
self.assertEqual(code, 0)
self.assertEqual(json.loads(out)["counts"], report["counts"])
self.assertIn("tracekit audit is experimental", err)
text = audit.format_report(report)
self.assertTrue(text.startswith("a replay against today's policy, not a signed record\n"))
self.assertIn("2 tool calls in 1 session: Tracekit would have denied 1 and held 0 for a person; it would have "
"flagged 0 and allowed 1.\n", text)
self.assertIn(" TK-D004: 1 (recursive delete of root or home)\n", text) # what each rule is
self.assertIn(" deny: 1\n", text)
self.assertIn("Bash deny", text)
self.assertNotIn("Bash allow", text) # the calls it would have stopped; --all lists every call
self.assertIn("Bash allow", audit.format_report(report, every=True))
self.assertIn("Bash allow", _run(str(self.root), "--all")[1])
self.assertEqual(_run(str(self.root), "--since", "soon")[0], 2)
code, _, err = _run(str(self.root / "missing"))
self.assertEqual(code, 2)
self.assertIn("no such folder", err)

def test_a_time_utc_cannot_hold_is_a_call_without_a_time(self):
line = _call("Bash", {"command": "ls"})
_write_jsonl(self.root / "s.jsonl", [{**line, "timestamp": "0001-01-01T00:00:00+05:00"}])
self.assertEqual([c["time"] for c in audit.audit(str(self.root))["calls"]], [None])

def test_control_characters_from_a_transcript_are_printed_as_text(self):
_write_jsonl(self.root / "s.jsonl", [_call("Bash\x1b[2K", {"command": "ls"})])
text = audit.format_report(audit.audit(str(self.root)), every=True)
self.assertNotIn("\x1b", text)
self.assertIn("\\u001b", text)


if __name__ == "__main__":
unittest.main()
Loading
Loading