Skip to content

tracekit audit: replay your coding agents' past tool calls through the policy and report what it would have held #490

Closed

Description

@LoopGlitch26

Coding agents keep their own transcripts on disk. Someone deciding whether to install Tracekit has a fair question: what would it have stopped last week? Today the only way to find out is to install it and wait.

Task: a read-only command, tracekit audit, that reads the tool calls in local coding-agent transcripts, decides each one with the policy engine, and prints what would have been held or denied.

  • Claude Code first: ~/.claude/projects/*/*.jsonl, one JSON object per line, with tool_use blocks in assistant lines. tracekit/tailer.py already reads this format (_report). Reuse its reading, not its signer calls. Codex (~/.codex/sessions/) can follow in a second PR.
  • Deciding: use tracekit.policy2.engine.Engine with a pack chosen by --policy. The default is the dev pack (tracekit/signer/service.py's DEFAULT_POLICY). Don't start a signer and don't write anything.
  • Output: a count per verdict and rule id, with the session and time of each call. Show the command text only with --show, with secrets redacted (tracekit.privacy.redact_text). Add --json for scripts and --since 7d.
  • Local only: no network, no upload, no telemetry. Read as the current user, and count and skip any file it can't read. Don't follow symlinks out of the transcript folders.
  • Say what it is: print "a replay against today's policy, not a signed record". Tracekit never recorded these calls, so nothing here is evidence.

Tests go in tests/test_audit.py. Use small synthetic transcripts in a temp folder, never a real one, covering:

  • a held call;
  • a denied call;
  • an allowed call;
  • a malformed line, which is skipped and counted;
  • an unreadable file.

Docs:

  • a new docs/audit.md;
  • a docs/limits.md row saying a replay sees only the agent's tool calls, and that a transcript the agent could edit is not evidence.

Done when:

  • tracekit audit on the test folder prints the counts above, and --json gives the same numbers;
  • nothing is written outside the temp folder;
  • make lint and make test pass.

Comment here to claim it. Commits need a DCO sign-off (git commit -s).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesthelp wantedExtra attention is needed

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions