Skip to content

Field report: Claude Code, 7 days on macOS #815

Open

Description

@aj-codespy

The output of tracekit report --share or tracekit audit --share

Tracekit 1.5.3 field report (tracekit report --share): 7 days
Integrity: the store verified, 408 of 408 runs verify
Runs that verify, by agent: claude-code 408
Tool calls: 25,746   allow 17,244, flag 8,382, ask 76, deny 44
Holds: 70   expired 63, approved · ran 6, not answered yet 1   (6 of the agent's retries of a held call, counted with it)
Gaps (signed records that something may be missing): 3,910   reconcile.fabricated 3,703, tailer_lost 121, reconcile.hook_missing 42, decided_not_completed 41, reconcile.args_mismatch 2, reconcile.result_without_call 1
Denied and flagged calls, by rule (one call can match several rules):
  deny TK-D002 34
  deny TK-D003 3
  deny TK-D010 3
  deny TK-D007 2
  deny TK-D008 2
  deny TK-SHELL-PARSE 2 (not placed)
  deny TK-F001 34 (also matched, not why)
  deny TK-F012 33 (also matched, not why)
  deny TK-F013 33 (also matched, not why)
  deny TK-F003 4 (also matched, not why)
  deny TK-F004 3 (also matched, not why)
  deny TK-A046 2 (also matched, not why)
  deny TK-F005 2 (also matched, not why)
  deny TK-F007 1 (also matched, not why)
  deny TK-F014 1 (also matched, not why)
  flag TK-F006 2,821
  flag TK-UNKNOWN-TOOL 2,634 (not placed)
  flag TK-F002 2,066
  flag TK-F012 1,831
  flag TK-F011 458
  flag TK-F003 249
  flag TK-F005 214
  flag TK-F008 142
  flag TK-F004 131
  flag TK-F001 127
  flag TK-F013 121
  flag TK-F015 42
  flag TK-F009 41
  flag TK-OVERSIZE 17 (not placed)
  flag TK-F007 2
  flag TK-F016 2
  flag TK-F017 1

Which agent or framework, and what kind of work

Claude Code 2.1.323, mostly the desktop app's Code tab, with several sessions in parallel. The work was a mix: web app and website development, Python and Go backends, and open-source contributions (this one included: the OpenHands harness, #563/#662).

How long, and on what

7 days (6–13 Nov) on a macOS 27 laptop (Apple silicon). Tracekit 1.5.3, installed with uv tool install (Python 3.12), tracekit init --dev --v2.

Anything you'd like to tell us (optional)

What went well: all 408 runs verify, and the hook never got in the way of a normal call. The denies I saw were right: a force-push (TK-D003) was blocked, and I pushed it myself, as the message said to. tracekit audit --since 7d beforehand was a good way to see what to expect.

Holds mostly expired (63 of 70). In the desktop app, a held call returns at once with the approve command, and the agent usually reworded the call or moved on rather than waiting. So most holds were never answered. That isn't wrong, but a shorter "approve this, or tell me to skip it" in the agent's message might get more of them answered.

Gaps: 3,703 reconcile.fabricated. That's most of the gaps, and I don't think the agent fabricated anything. My guess is calls the hook didn't see: parallel sessions in the desktop app, or sub-agents (Task). I can help narrow it down if you tell me what to look at.

False alarms I hit:

  • TK-SHELL-PARSE holds on ordinary commands that use shell arithmetic ($((…))) or a variable for a path (cd $DIR && …). Writing the command out plainly got through, as the message suggests, so it's friction rather than a blocker.
  • TK-D007 ("touching Tracekit's own files or signer") denied a python -c "import tracekit.signer…" run inside a dev checkout of the Tracekit repo. That's the repo's source, not the installed signer. It's probably rare outside Tracekit contributors, but working on Tracekit under Tracekit trips it.

Before you post

  • I removed anything private (secrets, hostnames, customer or project names, paths) from this report.

Activity

  1. LoopGlitch26 commented on Nov 13, 2026

    @LoopGlitch26
    Member

    Thank you. This is the first outside week we have, and it's exactly the kind of report that changes what we work on.

    The 3,703 reconcile.fabricated gaps are ours, not your agent's. We measured the same thing on our own Claude Code store. Nearly all of them are tool calls whose transcript lines the tailer never recorded, so reconcile can't match them. It has three causes:

    • the tailer stops partway through a run and isn't restarted;
    • the tailer gives up while the run is still open (your 121 tailer_lost fit this);
    • workflow and sub-agent transcripts are only partly read.

    Parallel sessions and Task sub-agents make all three more common, as you guessed. The fix is starting now; I'll link the PR here. Then a re-run of tracekit audit/report on your setup would be the best check of it, if you're willing.

    On the rest:

    • Holds that expire: agreed. A shorter "approve this, or tell me to skip it" in the agent's message is a good idea. I'll open it as an issue.
    • $((…)) and cd $DIR && … held by TK-SHELL-PARSE: noted. They're on our list for the shell reader, which holds what it can't fully read on purpose, so this needs care rather than a quick loosening.
    • TK-D007 on python -c "import tracekit.signer…" in a Tracekit checkout: right, that's a false alarm for anyone working on Tracekit itself. I'll open it as an issue too.

    The flag counts (TK-F006, TK-UNKNOWN-TOOL, TK-F002) also tell us where flags are noisiest on real work, which helps.

  2. LoopGlitch26 commented on Nov 13, 2026

    @LoopGlitch26
    Member

    Two follow-ups:

    • Holds that expire: opened as Holds: ask the person "approve or skip?" so held calls get answered (63 of 70 expired in #815) #816.
    • TK-D007: I looked closer and I'm taking back "false alarm". In a dev checkout, Tracekit is usually an editable install, so the tracekit.signer you import is the code that runs and signs the dev signer. The policy also can't trust the working directory, because the agent chooses it. So it can't tell a contributor's quick check apart from an agent reaching into its own signer. Code that only reads the RPC tables, reconcile.compared/model_names, or load_policy is allowed. Anything more is denied on purpose, as the TK-D007 row in docs/limits.md says. When working on Tracekit itself, run that code through the tests. So no issue for this one.

    The fix for the fabricated gaps is still in progress. I'll link the PR here.

  3. LoopGlitch26 commented on Nov 13, 2026

    @LoopGlitch26
    Member

    The fix for the fabricated gaps is #818.

  4. aj-codespy commented on Nov 13, 2026

    @aj-codespy
    ContributorAuthor

    @LoopGlitch26 Thanks for digging into it so quickly, and for opening #816.

    Yes, happy to re-run it. Once #818 is merged I'll update, keep running Tracekit on the same setup (parallel desktop sessions and Task sub-agents), and post tracekit audit/tracekit report --share here so you can compare the reconcile.fabricated and tailer_lost counts with this week's.

    On TK-D007: that makes sense. With an editable install that import really is the signer that runs, so I'll go through the tests for that.

  5. LoopGlitch26 commented on Nov 13, 2026

    @LoopGlitch26
    Member

    The fix for the fabricated gaps is merged (#818). A transcript reader that stopped is now started again after one signed gap and reads on from where it stopped. One that went quiet pauses instead of giving up after 2 hours, and the next event starts it again. On a scripted run shaped like your setup (parallel sessions, a sub-agent, a workflow agent and a killed reader), false reconcile.fabricated dropped from 44 to 0, and the planted real gaps were all still caught. What's not covered yet: calls Claude Code runs without a PreToolUse hook, and calls decided in another run of the same session. Your 42 hook_missing are those.

    To re-run on the same setup, either wait for 1.7.0 (soon), or install main now:

    uv tool install --force "tracekit-ai @ git+https://github.com/Cygnux-Labs/Tracekit@main"
    

    then restart your agent sessions so the hooks pick it up.

    One exact ask, since you offered. Your report had 2,634 TK-UNKNOWN-TOOL flags, and our own week has 130. Could you post the tool names behind them, names and counts only, with any private MCP server names replaced?

    tracekit audit --since 7d --json | jq -r '.calls[] | select(.rule_ids | index("TK-UNKNOWN-TOOL")) | .tool' | sort | uniq -c | sort -rn | head -20
    

    That tells us which tools to map, so they get decided instead of flagged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions