diff --git a/docs/audit.md b/docs/audit.md new file mode 100644 index 000000000..5497c8c02 --- /dev/null +++ b/docs/audit.md @@ -0,0 +1,65 @@ +# `tracekit audit` + +`tracekit audit` is a read-only command that replays the tool calls found in local coding-agent transcripts through the policy engine and reports what the current policy would have held or denied. + +It does not start a signer, does not write anything, and does not make network calls. It is a replay against today's policy, not a signed record. Tracekit never recorded these calls, so nothing here is evidence. + +It is experimental ([support](support.md)): it works and is tested, but may change. + +## Usage + +``` +tracekit audit [ROOT] [--policy PATH] [--since 7d] [--show] [--all] [--json] +``` + +``` +$ tracekit audit --since 7d --show +a replay against today's policy, not a signed record +policy: …/tracekit/policy2/packs/dev.yaml (sha256:…) + +1204 tool calls in 31 sessions since 7d: Tracekit would have denied 2 and held 9 for a person; it would have flagged 57 and allowed 1136. +… +by rule: + TK-F003: 41 (destructive) + TK-A002: 3 (deleting a remote Git ref) + TK-D004: 1 (recursive delete of root or home) +… +denied and held calls (--all for every call): + 4f1c… 2026-10-28T14:02:11+00:00 Bash deny rules=TK-D004,TK-F003 text='Bash rm -rf ~/' +``` + +(The numbers above are an example, not a measurement.) + +- `ROOT` defaults to `~/.claude/projects`; a folder that doesn't exist is an error (exit 2). +- `--policy` selects a policy file (default: the dev pack, `tracekit/policy2/packs/dev.yaml`). +- `--since` limits the replay (e.g. `7d`, `24h`, `30m`, or an ISO-8601 time, UTC unless it gives an offset); a call + without a timestamp is kept. +- `--show` includes each call as an approver sees it: its secrets redacted as the signer redacts them (`.env` content + too), cut to 72 characters. +- `--all` lists every call; without it, only the calls Tracekit would have denied or held for a person. +- `--json` emits a machine-readable report, with every call. + +The report always begins with the disclaimer: "a replay against today's policy, not a signed record". + +## What it reads + +Every `.jsonl` file under ROOT, at any depth: each Claude Code session's transcript, and its subagents' files +(`/subagents/*.jsonl`), which count under that session. Each line is a JSON object; assistant messages may contain `tool_use` blocks. The reader +reuses `tracekit.hook.transcript_entries` and the policy engine from `tracekit.policy2.engine`. + +It follows no symlink. Files and folders it can't read are counted and skipped, and so is a line that isn't JSON. + +## Output + +- One line first: how many calls in how many sessions, and how many Tracekit would have denied, held, flagged and + allowed. +- Counts per verdict (`deny`, `ask` / held, `flag`, `allow`). A call the policy engine fails on counts as a deny + (`TK-ENGINE-ERROR`), as the signer would decide it. +- Counts per rule id, most first, each with what the rule is (its `reason` or `label`, or the viewer's words for the + engine's own rules). +- The session and timestamp of each denied or held call (every call with `--all`), and its redacted text with + `--show`. + +## Limitations + +It sees only the tool calls the transcript holds, decided by today's policy. A transcript the agent could edit is not evidence. See [limits](limits.md#audit-tracekit-audit). diff --git a/docs/limits.md b/docs/limits.md index c1031ed90..994d53d59 100644 --- a/docs/limits.md +++ b/docs/limits.md @@ -773,6 +773,15 @@ What Tracekit guarantees and doesn't, per deployment, on one page: [guarantees]( | Before it places or counts a run, it checks every record of the store against the newest checkpoint, in every tenant (`--tenant` too) and the signer's own runs: its hash is its event's, it is filed under the run its event names, and the hashes make the checkpoint's root (3.6 seconds for a store of 246,344 records and 845 runs on a laptop, whatever `--since` or `--tenant`). A run holding an edited record is listed `FAILED` whatever its tenant and times, a signer's run or another tenant's included; when the hashes don't make the root (a hash was rewritten too), the store and every run fail; and the command exits 1. Records newer than the checkpoint are not checked against it and place no run: a run whose covered records all fall before the window isn't listed, even when newer records of it fall in the window. The report says how many records of the store are newer, and a run no checkpoint covers yet is `PENDING`. | The store files a record under the tenant, run id and time the record holds, and only the signed checkpoint vouches for them; in dev mode the agent's user can edit the store. | Run it again after the next checkpoint (the signer writes one within seconds); for a `FAILED` run, `tracekit view` shows its first failing check. | [viewer](viewer.md#weekly-report) | | It reads the v2 signer's store only, not the v1 ledger (`tracekitd`), why logs or bundles. It summarises no signer's run (its key records, content reads, refused writes, a witness that didn't countersign): it checks their records and lists one only when it fails. | It summarises the agents' runs of the store `tracekit view` verifies. | `tracekit view` shows the signer's own rows; `tracekit view --ledger` the v1 ledger. | [viewer](viewer.md#laptop-viewer) | +## Audit (`tracekit audit`) + +| Limit | Why it is so | What to do | Details | +|---|---|---|---| +| It reads Claude Code's transcripts only, and sees only the tool calls they hold: nothing the agent did outside a tool call, and nothing from another harness (Codex, Gemini CLI). | Each harness writes its transcripts in its own format; Claude Code's is the one it reads. | For what an agent does from now on, record it with Tracekit (`tracekit init --dev --v2`). | [audit](audit.md) | +| Its counts are evidence of nothing: the agent could have edited the transcript, and each call is decided by today's policy, not the one in force when it ran. | Tracekit never recorded these calls; only a signed record shows what happened. | Use it to see what Tracekit would hold; for evidence, record runs with the signer and export bundles. | [audit](audit.md) | +| It can count fewer holds than the signer would: each call is decided on its own, without the run's state, so `from: untrusted` rules never match (there is no provenance), and detect sequences, limits and the person's prompt constraints don't apply. | It reads a transcript, not a run; that state exists only in the signer. | Treat its counts as a floor; the signer's own decisions are the ones that hold. | [audit](audit.md) | +| It keeps every call it reads in memory (about 0.5 KB each; `--show` text is cut to 72 characters), so a very large transcript folder takes memory in proportion. | The JSON report and `--all` list every call, so it keeps them all until the counts are done. | Use `--since` to replay a shorter window. | [audit](audit.md) | + ## Install | Limit | Why it is so | What to do | Details | diff --git a/docs/support.md b/docs/support.md index 3043e4d8a..4b47c1c1c 100644 --- a/docs/support.md +++ b/docs/support.md @@ -30,6 +30,7 @@ experimental. | `tracekit why`: investigation, replay, signed findings | [why](why.md) | | `tracekit graph`: graph and report built from v2 bundles | [graph](graph.md) | | `tracekit report`: a week of a signer's store on one page, as text or JSON | [viewer](viewer.md#weekly-report) | +| `tracekit audit`: Claude Code transcripts replayed through the policy, read-only and not a record | [audit](audit.md) | | Detection and containment: sequences and limits, holds, quarantine, suspensions, incidents, the webhook outbox | [incidents](incidents.md), [webhooks](webhooks.md) | | Argument provenance (`untrusted`, `from_untrusted`) | [policy](policy-v2.md#provenance-untrusted-and-from-untrusted) | | Readable reasoning: what the model said, recorded by the transcript tailer, the coding-agent hooks, the framework integrations and the LLM gateway | [reasoning](reasoning.md) | diff --git a/scripts/wiki.py b/scripts/wiki.py index 9af419f3f..3ecfe3408 100644 --- a/scripts/wiki.py +++ b/scripts/wiki.py @@ -41,8 +41,8 @@ "concepts/deployment.md", "concepts/storage-and-performance.md", "concepts/integrations.md", "concepts/security-model.md", "architecture.md", "why-architecture.md"]), ("Integrations", ["integrations.md", "adapters.md", "otel.md", "remote-ingest.md", "integrations/*.md"]), - ("Policy and approvals", ["policy-v2.md", "approvals.md", "pcm.md", "findings.md", "reasoning.md", "incidents.md", - "webhooks.md"]), + ("Policy and approvals", ["policy-v2.md", "approvals.md", "pcm.md", "findings.md", "audit.md", "reasoning.md", + "incidents.md", "webhooks.md"]), ("Deploy and operate", ["deploy.md", "deploy-container.md", "deploy-kubernetes.md", "deploy-compose.md", "identity.md", "executor.md", "a2a.md", "c2c.md", "mcp-gateway.md", "gateway.md", "egress.md", "witnesses.md", "monitor.md", "issuer.md", "observability.md", "viewer.md", "horizon.md", "viewer-1.4.md", diff --git a/tests/test_audit.py b/tests/test_audit.py new file mode 100644 index 000000000..7d8d1f836 --- /dev/null +++ b/tests/test_audit.py @@ -0,0 +1,153 @@ +"""Tests for `tracekit audit` (issue #490).""" +from __future__ import annotations + +import contextlib +import io +import json +import os +import tempfile +import unittest +from datetime import datetime, timedelta, timezone +from pathlib import Path +from unittest import mock + +from tracekit import audit, cli + + +def _call(tool, args, days_ago=0): + """An assistant line with one tool use, as Claude Code writes it.""" + at = datetime.now(timezone.utc) - timedelta(days=days_ago) + return {"type": "assistant", "timestamp": at.isoformat().replace("+00:00", "Z"), + "message": {"content": [{"type": "tool_use", "name": tool, "input": args}]}} + + +def _write_jsonl(path: Path, lines: list[dict | str]): + path.parent.mkdir(parents=True, exist_ok=True) + with path.open("w", encoding="utf-8", newline="\n") as f: + for line in lines: + f.write((line if isinstance(line, str) else json.dumps(line)) + "\n") + + +def _run(*argv): + out, err = io.StringIO(), io.StringIO() + with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err): + code = cli.main(["audit", *argv]) + return code, out.getvalue(), err.getvalue() + + +class TestAudit(unittest.TestCase): + def setUp(self): + self.tmp = tempfile.TemporaryDirectory() + self.root = Path(self.tmp.name) + + def tearDown(self): + self.tmp.cleanup() + + def test_held_denied_allowed_and_malformed(self): + _write_jsonl(self.root / "proj" / "session-abc.jsonl", [ + _call("Bash", {"command": "rm -rf /"}), _call("Bash", {"command": "terraform destroy -auto-approve"}), + _call("Bash", {"command": "ls -la"}), "this is not json", {"type": "user", "message": {"content": "hi"}}]) + report = audit.audit(str(self.root)) + self.assertEqual(report["counts"], {"deny": 1, "ask": 1, "flag": 0, "allow": 1}) + self.assertIn("TK-D004", report["by_rule"]) + self.assertIn("TK-A001", report["by_rule"]) + self.assertEqual(report["skipped_malformed"], 1) + self.assertEqual({c["session"] for c in report["calls"]}, {"session-abc"}) + self.assertTrue(all(c["time"] for c in report["calls"])) + self.assertNotIn("text", report["calls"][0]) # the command text only with --show + self.assertIn("not a signed record", report["disclaimer"]) + + def test_a_subagents_calls_count_under_its_session(self): + _write_jsonl(self.root / "proj" / "s1.jsonl", [_call("Bash", {"command": "ls"})]) + _write_jsonl(self.root / "proj" / "s1" / "subagents" / "agent-a.jsonl", [_call("Bash", {"command": "ls"})]) + report = audit.audit(str(self.root)) + self.assertEqual((report["sessions"], {c["session"] for c in report["calls"]}), (1, {"s1"})) + + def test_a_call_the_engine_fails_on_is_a_deny_as_the_signer_decides_it(self): + _write_jsonl(self.root / "s.jsonl", [_call("Bash", {"command": "ls"})]) + with mock.patch("tracekit.policy2.engine.Engine.decide", side_effect=RuntimeError("boom")): + report = audit.audit(str(self.root)) + self.assertEqual(report["counts"]["deny"], 1) + self.assertEqual(report["by_rule"], {"TK-ENGINE-ERROR": 1}) + self.assertEqual(report["skipped_malformed"], 0) + + def test_an_unreadable_file_is_counted_and_skipped(self): + bad = self.root / "proj" / "bad.jsonl" + _write_jsonl(bad, [_call("Bash", {"command": "rm -rf /"})]) + bad.chmod(0) + if os.access(bad, os.R_OK): # root, or a file system that ignores the mode (Windows) + self.skipTest("the file stays readable here") + report = audit.audit(str(self.root)) + self.assertEqual(len(report["unreadable"]), 1) + self.assertEqual(report["calls"], []) + + def test_no_symlink_is_followed_but_a_root_reached_through_one_is_read(self): + with tempfile.TemporaryDirectory() as outside: + real = Path(outside) / "other.jsonl" + _write_jsonl(real, [_call("Bash", {"command": "rm -rf /"})]) + try: + (self.root / "link.jsonl").symlink_to(real) + (self.root / "linkdir").symlink_to(outside, target_is_directory=True) + except OSError: # Windows without the privilege + self.skipTest("no symlinks here") + self.assertEqual(audit.audit(str(self.root))["calls"], []) + self.assertEqual(audit.audit(str(self.root / "linkdir"))["counts"]["deny"], 1) # a root that is a symlink + + def test_since_keeps_only_newer_calls(self): + _write_jsonl(self.root / "s.jsonl", [_call("Bash", {"command": "echo old"}, days_ago=30), + _call("Bash", {"command": "echo new"}, days_ago=1)]) + ten_days_ago = datetime.now(timezone.utc) - timedelta(days=10) + for since in ("10d", "240h", ten_days_ago.astimezone(timezone(timedelta(hours=5))).isoformat(), + ten_days_ago.replace(tzinfo=None).isoformat()): # an offset, and none (UTC) + report = audit.audit(str(self.root), since=since, show=True) + self.assertEqual([c["text"] for c in report["calls"]], ["Bash echo new"], since) + with self.assertRaises(ValueError): + audit.audit(str(self.root), since="last week") + + def test_show_redacts_secrets_as_the_signer_does(self): + key = "sk-ant-" + "a1B2c3D4e5F6g7H8i9J0" * 2 + _write_jsonl(self.root / "s.jsonl", [ + _call("Bash", {"command": f"export ANTHROPIC_API_KEY={key}"}), + _call("Bash", {"command": "cat > .env <<'EOF'\nSTRIPE_WEBHOOK=whsec_abc123\nEOF"})]) + key_text, env_text = (c["text"] for c in audit.audit(str(self.root), show=True)["calls"]) + self.assertNotIn(key, key_text) + self.assertIn("export ANTHROPIC_API_KEY=", key_text) + self.assertNotIn("whsec_abc123", env_text) # a .env write: every value, as the signer redacts it + self.assertIn("STRIPE_WEBHOOK=", env_text) + + def test_the_command_prints_the_report_and_json_matches_it(self): + _write_jsonl(self.root / "s.jsonl", [_call("Bash", {"command": "rm -rf /"}), _call("Bash", {"command": "ls"})]) + report = audit.audit(str(self.root)) + code, out, err = _run(str(self.root), "--json") + self.assertEqual(code, 0) + self.assertEqual(json.loads(out)["counts"], report["counts"]) + self.assertIn("tracekit audit is experimental", err) + text = audit.format_report(report) + self.assertTrue(text.startswith("a replay against today's policy, not a signed record\n")) + self.assertIn("2 tool calls in 1 session: Tracekit would have denied 1 and held 0 for a person; it would have " + "flagged 0 and allowed 1.\n", text) + self.assertIn(" TK-D004: 1 (recursive delete of root or home)\n", text) # what each rule is + self.assertIn(" deny: 1\n", text) + self.assertIn("Bash deny", text) + self.assertNotIn("Bash allow", text) # the calls it would have stopped; --all lists every call + self.assertIn("Bash allow", audit.format_report(report, every=True)) + self.assertIn("Bash allow", _run(str(self.root), "--all")[1]) + self.assertEqual(_run(str(self.root), "--since", "soon")[0], 2) + code, _, err = _run(str(self.root / "missing")) + self.assertEqual(code, 2) + self.assertIn("no such folder", err) + + def test_a_time_utc_cannot_hold_is_a_call_without_a_time(self): + line = _call("Bash", {"command": "ls"}) + _write_jsonl(self.root / "s.jsonl", [{**line, "timestamp": "0001-01-01T00:00:00+05:00"}]) + self.assertEqual([c["time"] for c in audit.audit(str(self.root))["calls"]], [None]) + + def test_control_characters_from_a_transcript_are_printed_as_text(self): + _write_jsonl(self.root / "s.jsonl", [_call("Bash\x1b[2K", {"command": "ls"})]) + text = audit.format_report(audit.audit(str(self.root)), every=True) + self.assertNotIn("\x1b", text) + self.assertIn("\\u001b", text) + + +if __name__ == "__main__": + unittest.main() diff --git a/tracekit/audit.py b/tracekit/audit.py new file mode 100644 index 000000000..46d29d6b7 --- /dev/null +++ b/tracekit/audit.py @@ -0,0 +1,251 @@ +"""tracekit audit: replay local coding-agent transcripts through the policy engine. + +Read-only. Never starts a signer, never writes, never follows a symlink. A replay against today's policy, not a signed +record. +""" +from __future__ import annotations + +import argparse +import json +import os +import stat +import sys +import time +from datetime import datetime, timezone +from typing import Any + +from .core import scrub +from .hook import transcript_entries +from .report import call_text +from .signer.service import DEFAULT_POLICY, _dotenv, _escaped, _shown, load_policy +from .view import ENGINE_RULES + +DEFAULT_ROOT = os.path.join("~", ".claude", "projects") + + +def _parse_since(s: str) -> float: + """Return epoch seconds for --since (e.g. 7d, 24h, 30m, or an ISO date or time, UTC unless it says).""" + s = s.strip().lower() + if s.endswith("d") and s[:-1].isdigit(): + return time.time() - int(s[:-1]) * 86400 + if s.endswith("h") and s[:-1].isdigit(): + return time.time() - int(s[:-1]) * 3600 + if s.endswith("m") and s[:-1].isdigit(): + return time.time() - int(s[:-1]) * 60 + try: + dt = datetime.fromisoformat(s.replace("z", "+00:00")) + if dt.tzinfo is None: + dt = dt.replace(tzinfo=timezone.utc) + return dt.timestamp() + except ValueError: + raise ValueError(f"unrecognised --since {s!r} (use Nd, Nh, Nm or ISO-8601)") + + +def _session_from_path(path: str) -> str: + """Best-effort session id from a Claude Code transcript path: a subagent's file (/subagents/agent-*.jsonl) + counts under its session.""" + folder = os.path.dirname(path) + if os.path.basename(folder) == "subagents": + return os.path.basename(os.path.dirname(folder)) + base = os.path.basename(path) + if base.endswith(".jsonl"): + return base[:-6] + return base + + +def _entry_time(entry: dict) -> datetime | None: + """Time of a transcript entry if present (Claude Code uses 'timestamp'), UTC unless it says.""" + ts = entry.get("timestamp") + if isinstance(ts, str): + try: + dt = datetime.fromisoformat(ts.replace("Z", "+00:00")) + return (dt if dt.tzinfo else dt.replace(tzinfo=timezone.utc)).astimezone(timezone.utc) + except (ValueError, OverflowError): # not a time, or one UTC can't hold (0001-01-01T00:00+05:00) + pass + return None + + +def _safe_open(path: str) -> tuple[Any, str | None]: + """Open a regular file without following a symlink (as graph._secure_source does). Returns (file, error).""" + try: + seen = os.lstat(path) + except OSError as e: + return None, f"cannot stat: {e}" + if not stat.S_ISREG(seen.st_mode): + return None, "not a regular file or is a symlink" + flags = os.O_RDONLY | getattr(os, "O_NONBLOCK", 0) | getattr(os, "O_NOFOLLOW", 0) | getattr(os, "O_BINARY", 0) + try: + fd = os.open(path, flags) + except OSError as e: + return None, f"cannot open: {e}" + opened = os.fstat(fd) + if not stat.S_ISREG(opened.st_mode) or (opened.st_dev, opened.st_ino) != (seen.st_dev, seen.st_ino): + os.close(fd) + return None, "changed while it was opened" + return os.fdopen(fd, "rb"), None + + +def _iter_transcripts(root: str, unreadable: list): + """Yield (path, session) for each .jsonl under root, at any depth (a session's subagent files too). Follows no + symlink; a folder it can't list goes into `unreadable`.""" + root = os.path.realpath(root) # a root reached through a symlink (macOS /var, a linked ~/.claude) is still the root + + def failed(e): + unreadable.append({"path": e.filename, "error": f"cannot list: {e}"}) + for dirpath, dirnames, filenames in os.walk(root, onerror=failed, followlinks=False): + dirnames[:] = [d for d in dirnames if not os.path.islink(os.path.join(dirpath, d))] + for name in filenames: + path = os.path.join(dirpath, name) + if not name.endswith(".jsonl") or os.path.islink(path): + continue + try: # stay inside root: a Windows directory junction is no symlink to islink, and the walk enters it + if os.path.commonpath([root, os.path.realpath(path)]) != root: + continue + except ValueError: # another drive + continue + yield path, _session_from_path(path) + + +def audit(root: str, policy_path: str | None = None, since: str | None = None, show: bool = False) -> dict: + """Replay tool calls under root. Returns the report dict. + + Counts malformed lines and unreadable files. Never writes. + """ + since_ts = _parse_since(since) if since else 0.0 + engine = load_policy(policy_path or DEFAULT_POLICY) + + counts = {"allow": 0, "ask": 0, "deny": 0, "flag": 0} + by_rule: dict[str, int] = {} + # lean: every call is kept in memory (about 0.5 KB each; --show text is capped at report.CALL_WIDTH); stream the + # report's lines instead if a real transcript folder outgrows that + calls: list[dict] = [] + skipped_malformed = 0 + unreadable: list[dict] = [] + + for path, session in _iter_transcripts(root, unreadable): + f, err = _safe_open(path) + if err: + unreadable.append({"path": path, "error": err}) + continue + try: + for _n, entry in transcript_entries(f): + if entry is None: + skipped_malformed += 1 + continue + if entry.get("type") != "assistant": + continue + at = _entry_time(entry) + if at is not None and at.timestamp() < since_ts: + continue + msg = entry.get("message") if isinstance(entry.get("message"), dict) else {} + content = msg.get("content") + blocks = content if isinstance(content, list) else [] + for b in blocks: + if not isinstance(b, dict) or b.get("type") != "tool_use": + continue + tool = b.get("name") if isinstance(b.get("name"), str) else "" + args = b.get("input") if isinstance(b.get("input"), dict) else {} + try: + decision = engine.decide(tool, args) + except Exception: # the signer denies a call its engine fails on (signer.service._evaluate) + decision = {"verdict": "deny", "rule_ids": ["TK-ENGINE-ERROR"]} + verdict = decision.get("verdict", "allow") + counts[verdict] = counts.get(verdict, 0) + 1 + for rid in decision.get("rule_ids") or []: + by_rule[rid] = by_rule.get(rid, 0) + 1 + rec = { + "session": session, + "time": at.isoformat() if at else None, + "tool": tool, + "verdict": verdict, + "rule_ids": decision.get("rule_ids") or [], + } + if show: # what an approver sees: redacted as the signer redacts it (.env content too) + rec["text"] = call_text(tool, _shown(scrub(args), _dotenv(args))) + calls.append(rec) + except (OSError, RecursionError) as e: # a read that fails, or a line nested past what json.loads takes + unreadable.append({"path": path, "error": f"cannot read: {type(e).__name__}: {e}"}) + finally: + f.close() + + report = { + "disclaimer": "a replay against today's policy, not a signed record", + "policy": policy_path or DEFAULT_POLICY, + "policy_hash": engine.policy_hash, + "since": since, + "sessions": len({c["session"] for c in calls}), + "counts": counts, + "by_rule": by_rule, + "reasons": {**{i: ENGINE_RULES[i] for i in by_rule if i in ENGINE_RULES}, # what each rule is, as the viewer says it + **{r["id"]: r.get("reason") or r.get("label", "") for _, r, *_ in engine.rules if r["id"] in by_rule}}, + "skipped_malformed": skipped_malformed, + "unreadable": unreadable, + "calls": calls, + } + return report + + +def format_report(report: dict, as_json: bool = False, every: bool = False) -> str: + """The report as text: what Tracekit would have stopped first, then each denied or held call (every call with + `every`); or the whole report as JSON.""" + if as_json: + return json.dumps(report, indent=2, sort_keys=True) + n, s = report["counts"], report["sessions"] + total = sum(n.values()) + window = f" since {report['since']}" if report["since"] else "" + lines = [ + report["disclaimer"], + f"policy: {report['policy']} ({report['policy_hash']})", + "", + f"{total} tool call{'s' * (total != 1)} in {s} session{'s' * (s != 1)}{window}: Tracekit would have denied " + f"{n.get('deny', 0)} and held {n.get('ask', 0)} for a person; it would have flagged {n.get('flag', 0)} " + f"and allowed {n.get('allow', 0)}.", + "", + "counts:", + ] + for v in ("deny", "ask", "flag", "allow"): + lines.append(f" {v}: {n.get(v, 0)}") + if report["by_rule"]: + lines.append("") + lines.append("by rule:") + for rid, k in sorted(report["by_rule"].items(), key=lambda x: (-x[1], x[0])): + reason = report["reasons"].get(rid, "") + lines.append(f" {rid}: {k}" + (f" ({reason})" if reason else "")) + lines.append("") + lines.append(f"skipped malformed lines: {report['skipped_malformed']}") + lines.append(f"unreadable files: {len(report['unreadable'])}") + shown = [c for c in report["calls"] if every or c["verdict"] in ("deny", "ask")] + if shown: + lines.append("") + lines.append("calls:" if every else "denied and held calls (--all for every call):") + for c in shown: + t = c["time"] or "unknown-time" + extra = f" text={c['text']!r}" if "text" in c else "" + lines.append(f" {c['session']} {t} {c['tool']} {c['verdict']} rules={','.join(c['rule_ids'])}{extra}") + # a transcript the agent wrote can hold control characters: print them as text, as tracekit report does + return "\n".join(_escaped(line) for line in lines) + "\n" + + +def main(argv=None): + ap = argparse.ArgumentParser(prog="tracekit audit", description="replay the tool calls in your coding agents' " + "transcripts (Claude Code) through the policy and count what it would have held or " + "denied. Read-only, local, and a replay against today's policy, not a signed record.") + ap.add_argument("root", nargs="?", default=DEFAULT_ROOT, help="the transcripts' folder (default: ~/.claude/projects)") + ap.add_argument("--policy", help="a policy file (default: the dev pack, tracekit/policy2/packs/dev.yaml)") + ap.add_argument("--since", help="7d, 24h, 30m, or an ISO date or time (UTC unless it says)") + ap.add_argument("--show", action="store_true", help="also print each call, with secrets redacted") + ap.add_argument("--all", action="store_true", help="list every call, not only the denied and held ones") + ap.add_argument("--json", action="store_true", help="print the report as JSON (every call)") + a = ap.parse_args(argv) + root = os.path.expanduser(a.root) + if not os.path.isdir(root): + print(f"tracekit audit: no such folder: {root} (Claude Code keeps its transcripts in ~/.claude/projects)", + file=sys.stderr) + return 2 + try: + report = audit(root, a.policy, a.since, a.show) + except (ValueError, OSError) as e: # an unreadable --since; a policy file that's missing or doesn't lint + print(f"tracekit audit: {e}", file=sys.stderr) + return 2 + sys.stdout.write(format_report(report, as_json=a.json, every=a.all)) + return 0 diff --git a/tracekit/cli.py b/tracekit/cli.py index 8aff0b0e4..24a33de77 100644 --- a/tracekit/cli.py +++ b/tracekit/cli.py @@ -43,7 +43,7 @@ def _signer_home(a): "why": "why.cli", "reviewer": "reviewer", "srt-decider": "srt_decider", "openshell-middleware": "openshell", "follow": "follow", "sensor": "sensor", "ingest": "ingest", "graph": "graph", "pcm": "pcm.cli", - "report": "report"} # subcommands with their own parsers + "report": "report", "audit": "audit"} # subcommands with their own parsers # delegated subcommands that are experimental (docs/support.md); otel has its own --experimental gate _EXPERIMENTAL = {"why": "tracekit why", "gateway": "the LLM gateway", "egress": "the egress observer", "executor": "the tool executor", "mcp-gateway": "the MCP gateway", "a2a-gateway": "the A2A gateway", @@ -55,7 +55,7 @@ def _signer_home(a): "analyze": "tracekit analyze", "observe": "tracekit observe (the v1 ledger)", "srt-decider": "the srt-proxy decider", "follow": "tracekit follow", "sensor": "the kernel sensor import", "graph": "tracekit graph", "pcm": "tracekit pcm", - "report": "tracekit report"} + "report": "tracekit report", "audit": "tracekit audit"} _MOVED = {"sql": "query", "proofpack": "proofpack", "causeway": "causeway"} # now separate packages under contrib/ @@ -245,6 +245,9 @@ def main(argv=None): p = sub.add_parser("report", help="what the agents did this week, on one page, from the v2 signer's store: runs, " "calls, holds, denies and flags by rule, gaps, integrity (--since, --json)", add_help=False) p.add_argument("rest", nargs=argparse.REMAINDER) + p = sub.add_parser("audit", help="replay your coding agents' past tool calls (Claude Code transcripts) through the " + "policy and count what it would have held or denied; read-only, not a signed record", add_help=False) + p.add_argument("rest", nargs=argparse.REMAINDER) p = sub.add_parser("why", help="causal investigation and replay of multi-agent runs: `why demo|record|serve|test|...`", add_help=False) p.add_argument("rest", nargs=argparse.REMAINDER)