Skip to content

Sign each Claude Code turn's usage as it happens; tips prefers signed counts (#865) - #871

Merged
LoopGlitch26 merged 5 commits into
mainfrom
agent/issue-865
Nov 15, 2026
Merged

LoopGlitch26 merged 5 commits into
mainfrom
agent/issue-865

Conversation

@LoopGlitch26

@LoopGlitch26 LoopGlitch26 commented Nov 15, 2026 •

Copy link
Copy Markdown
Member

Closes #865.

What changes

  • The transcript tailer signs each Claude Code turn's usage as it reads it. The tailer already signs every assistant line as a model.exchange response (model_event, exchange id = the message id). It now adds usage (input, output, cache-read and cache-write tokens, normalised by usage.from_anthropic, as the LLM gateway records them) once per message id in the run, only at the turn's final counts (a line with a stop_reason), on the message's last line a read takes in, and for a turn that stopped for a tool, on a tool_use line. The ids it signed go into the dev-mode progress log ([transcript, offset, digest, message id]), so a restarted tailer signs none again, and a subagent transcript that opens with a copy of the message that started it doesn't sign it twice.
  • tracekit tips prefers signed usage, from the right record only. A signed record takes a transcript turn's place, with its signed counts (the transcript keeps which agent took the turn, when, and the cache-lifetime split while it adds up), in three passes:
    1. by message id, only from the session's own run: a claude-code run whose records name a tool call of that session, signed from the turn's first line to 5 minutes after its last;
    2. by tool_use id, only from an LLM gateway's record (its record source), within 5 minutes;
    3. by identical counts within 5 minutes, compared with the counts the tailer signed.
      Where the gateway and the tailer both signed a turn, it counts once and the gateway's counts are kept (signed off the wire). Other records of one turn that disagree put none of their counts on it: each is refused and the turn keeps the counts it had, so a run can push a signed turn down to the transcript's counts but never put its own over another's. A gateway's record never counts for nothing: it never joins by message id (a transcript line can't claim its exchange id), on a conflict only the other records are refused, one that joins no turn in its window counts on its own, and two gateways that disagree both count, so a run can make tips count more, never hide what a gateway signed. Any other record of a transcript's turn is refused too: it counts for nothing, the transcript's counts stand, and sources.signed.refused, tracekit tips and the TIPS page say how many. An exchange id counts once in a run; across runs where no transcript holds it, once when the counts agree; when they don't, a gateway's record counts and the others are refused, and without one none counts. Each tip says where its counts came from: from, a line in the context tip's arithmetic, and src on each cache wait.
  • The viewer's token totals count a turn once: an exchange id once, and a turn the gateway and the tailer both signed (joined by tool_use id, or the same counts within 5 minutes, one turn per record) at the gateway's counts.
  • Docs: docs/privacy.md rows for the v2 gateway and tailer fields (usage, model, message id, tool use ids and names, tool_results_sent and the signed time in clear; the line only as a salted commitment, in the vault only when the signer keeps content) and what output counts reveal; docs/limits.md rows (signed at read time, made-up lines, the 25 MB check, the session's own run token) and the TIPS row; the TIPS paragraph in docs/viewer.md; one sentence in docs/coding-agents.md.

Design choice

The smallest path used what exists. model.exchange already has a usage field, model_event already takes it (RPC v23 unchanged), and the signer already writes it. So: no new event type, no schema change, no RPC contract change, nothing for the verifiers or the conformance kit. model_event never feeds the provenance index (only complete of untrusted tools and observe do), so usage numbers can't become trusted values there.

The issue pictured the hook reading new transcript bytes at each event. The v2 hook already starts a detached tailer per run that does exactly that (from an offset kept beside the session state, checked, with signed tailer_lost gaps for a broken transcript), so hook latency doesn't change: PreToolUse stayed at a 73-79 ms median in the user runs, with and without this change.

The record names no session: any run can call model_event with any message id and any counts. Tips therefore binds a record to a turn by its run, its time and, for the gateway, the gateway's own identity; it never lets one record's counts go over another's, and never drops a gateway's. What that can't close: any run that names one of the session's calls joins its turns (the agent's user can read the session's run token, and start runs of its own), so it can sign made-up counts for a turn no other run signed within the window (a limits row; the gateway is the answer where spend must not rest on the agent's word).

Findings from the user runs:

  • Final counts only: on a real 2.1.286 session, 714 of 8,415 messages had subagent lines written mid-stream (no stop_reason, output 16) and a final line later (output 243). Over 120 days here, 68,076 subagent messages of that build never got a final line (median output 8). Current builds (2.1.308, 2.1.311) write a stop_reason on every line; 17 of 88,119 messages in two weeks had none (interrupted turns). Those stay with the transcript tier.
  • The time bound works: a replay that kept the session's original line times (days old) had every record refused (109 of 109, said in refused); with the lines written live, all 109 joined.

Fail-first

Each test that a fix needs, run on the code before that fix: round 1 on main (f32c2fe), the 1st review's on 421d341, the 2nd review's on 2d13c6f, the final check's on 3bf3fca and e7a3e1a.

Test Fails on
test_tailer.py::Tailer::test_a_messages_usage_is_signed_once_on_its_last_line main
test_tailer.py::Tailer::test_a_turn_streamed_over_several_lines_is_signed_at_its_final_counts main
test_tailer.py::Tailer::test_a_message_read_over_two_wakes_and_its_subagents_copy_of_it_sign_its_usage_once main, and 421d341 (usage on the thinking line)
test_tailer.py::Tailer::test_a_turns_usage_is_signed_in_clear_and_its_text_only_as_a_commitment main
test_tailer.py::Tailer::test_usage_never_enters_the_provenance_index main
test_tailer.py::HookRestartsTheTailer::test_a_tailer_started_again_signs_no_usage_the_stopped_one_signed main
test_tips.py::Sources::test_signed_usage_counts_before_the_transcripts_and_each_tip_says_where_its_counts_came_from main
test_tips.py::Sources::test_a_turn_signed_twice_counts_once main
test_tips.py::Sources::test_a_session_the_gateway_signed_and_the_transcripts_hold_counts_once (changed: from is signed records) main
test_tips.py::Sources::test_a_turn_the_gateway_and_the_tailer_both_signed_counts_once_at_the_gateways_counts 421d341
test_tips.py::Sources::test_only_the_sessions_own_run_signs_its_turns_and_only_after_them (5 subtests) 421d341, all 5
test_tips.py::Page::test_the_read_line_says_what_counted_once_and_that_every_project_is_listed (changed) main, and 421d341
test_tips.py::Page::test_the_viewers_token_totals_count_a_turn_signed_twice_once 421d341
test_tips.py::Sources::test_records_of_one_turn_that_disagree_put_none_of_their_counts_on_it (forger after, forger first) 2d13c6f, both subtests
test_tips.py::Sources::test_an_exchange_id_two_runs_signed_with_other_counts_counts_for_neither 2d13c6f
test_tips.py::Page::test_the_viewers_token_totals_count_a_turn_the_gateway_and_the_tailer_signed_once 2d13c6f; with the twin turns added, 3bf3fca (3 turns, not 2)
test_tips.py::Sources::test_a_gateways_record_never_counts_for_nothing (a bot re-signs the gateway's ids; a moved transcript turn; two gateways disagree) 3bf3fca ((0, 10) for (5, 5))
test_tips.py::Sources::test_a_transcript_line_under_a_gateways_exchange_id_leaves_the_gateways_record_counted (the gateway's run named gateway, or claude-code) e7a3e1a, both subtests ((0, 5) for (5, 0))

Mutation check

One mutation at a time on the named file, the guarding tests run, the file written back. 31 of 31 killed.

Property Mutation
signed once by message id drop mid not in send.signed
never twice across tailers a restarted tailer starts with no signed ids
on the message's last line every line counts as last
final counts only sign a line with no stop_reason
a tool turn on its tool_use line drop the uses or stop != "tool_use" condition
content only as a commitment the record carries the line's raw sha256 instead of the salted HMAC commitment
not trusted in provenance also observe the usage as a trusted input
tips prefers signed keep the transcript's counts, labelled signed
tips' tier from names the agent's source, not its counts'
an exchange id once in a run drop the per-run dedupe
an exchange id once across runs drop the cross-run dedupe
gateway + tailer count once keep tailer-joined turns out of the count pass
the gateway's counts kept keep the tailer's counts over the gateway's
session's own run only any agent's run joins by message id
session's own run only a run that names no call of the session joins
after the turn a record signed before the turn joins
within the window a record signed long after the turn joins
tool_use id joins only a gateway a non-gateway record joins by tool_use id
said on the CLI tracekit tips doesn't say how many were refused
said on the page the TIPS page doesn't say how many were refused
viewer totals once the viewer counts a turn signed twice twice
disagreeing records take neither the first record's counts are taken
an id two runs signed with other counts the first run's counts are kept
viewer: the gateway's counts kept the first record's counts are kept
viewer: gateway and tailer join by counts no join by counts
viewer: gateway and tailer join by tool_use id no join by tool_use id
a gateway's record survives a collision it is refused like any other
a gateway's record outside its window counts it is refused
two gateways that disagree both count both are refused
viewer: one turn per record in the count join only the latest turn with those counts can be joined
a gateway's record never joins by message id pass 1 takes a gateway's record whose id a transcript line claims

User run

The hook run as Claude Code calls it (python -I -m tracekit.integrations.claude_code EVENT, payload on stdin: SessionStart, UserPromptSubmit, PreToolUse/PostToolUse for the read-only calls, Stop, SessionEnd) against a scratch dev signer (its own HOME, runtime dir and CLAUDE_CONFIG_DIR), with a real transcript copy from ~/.claude/projects appended in 25 chunks as Claude Code would write it and its subagents' transcripts appearing mid-session. The source transcripts were only read; the copies are deleted. Each replay ran on this branch and on main.

Claude Code 2.1.308 session (6 MB slice + 2 subagents) Claude Code 2.1.286 session (2,361 lines + 159 subagent files)
Bundle final, tracekit verify exit 0: "VERIFIED: 166 tool calls in 539 signed records" final, exit 0: "VERIFIED: 11368 tool calls in 30767 signed records"
Records with usage, main 0 0
Records with usage, branch 109 of 109 messages; each id once; each equal to its final transcript line 1,509: every turn with a final line, each once, each equal to it; the main session's 243 of 245 (2 interrupted turns have no final line); that build's subagent turns were written mid-stream only and stay unsigned
Hook latency (median) PreToolUse 73-75 ms, PostToolUse 82-93 ms, both sides alike PreToolUse 79 ms, PostToolUse 93 ms, both sides alike

tracekit tips --json on the same scratch stores and transcripts:

main branch
2.1.286 session: sources.signed.also {runs: 0, steps: 0} {runs: 1, steps: 1509} (first head)
context tip from Claude Code's transcripts (read-only) signed records ("Token counts of the 243 steps these figures rest on: 243 from signed records.")
2.1.308 slice, lines written live (this head) {0, 0}, cache tip from transcripts also {runs: 1, steps: 109}, refused: 0
2.1.308 slice, the lines' original times (this head) also {0, 0}, refused: 109: signed days after the lines, so the transcript's counts stand

Gate

PATH=<repo>/.venv/bin:$PATH PYTHONPATH=$PWD make test-ts lint build test in the worktree, on e7a3e1a: exit 0. TS 173 tests (161 pass, 12 skipped, 0 fail); pyflakes clean; sdist and wheel built, twine check passed; pytest 5332 passed, 447 skipped, 10 xfailed, 74,947 subtests passed (35 min). On this head (one line in tips._join and its test): make lint clean, and tests/test_tips.py, test_tailer.py, test_view.py, test_view_ui.py, test_docs.py: 434 passed, 1 skipped, 325 subtests passed. macOS only: CI's Linux run is the merge gate.

Limits and open points

  • Signed at read time: an edit to a transcript line, or a whole made-up line, written before the tailer reads it (about a second while a tailer runs; at the next hook event after a kill or park) is signed as written; the gateway path signs off the wire (docs/limits.md).
  • Any run that names one of the session's calls joins its turns (the agent's user can read the session's run token): it can sign made-up counts for a turn no other run signed within the window, and push a tailer-signed turn down to the transcript's counts by disagreeing, but not replace signed counts, and never make a gateway's record count for nothing (at worst a turn counts twice). A session whose turns call no tool joins no run and keeps the transcript's counts.
  • A turn with no stop_reason line is left unsigned (interrupted turns; older builds' streamed subagent lines).
  • A line the signer took while its answer was lost, or one a killed tailer sent but hadn't logged, is signed twice; tips and the viewer count an exchange id once. Two runs that sign one exchange id no transcript holds with other counts count for neither.
  • The signed usage has no cache lifetime and names no subagent: tips takes both from the transcript when it has it; without transcripts (--data-dir) a session's subagent turns count as its main session's (lean: in tips.signed_steps).
  • The record's time is the signer's; the transcript line's own timestamp is only in the commitment (a clear field would be a format change).
  • Not checked: the TIPS page in a browser (two sentences changed, covered by the page tests), Linux, system mode's tailer user.

…rs signed counts (#865)

The transcript tailer already signs every assistant line as a model.exchange
(model_event). It now also signs the message's usage (input, output,
cache-read and cache-write tokens, normalised as the LLM gateway records
them) once per message id in the run: on the message's last line a read
takes in, and only at its final counts (a line with a stop_reason). The ids
signed go into the dev-mode progress log, so a restarted tailer never signs
one again; a subagent transcript's copy of the message that started it is
not signed twice. No new event type, schema or RPC change: model.exchange
and model_event already carry `usage`, and model_event never feeds the
provenance index.

tracekit tips joins a signed step to its transcript message by message id
(as well as tool_use id and counts), takes the signed counts in the
transcript's place (keeping the transcript's cache-lifetime split while it
adds up), counts an exchange id once, and says where each figure's counts
came from: the tip's `from`, the context tip's arithmetic line, each
wait's `src`.

Docs: privacy.md rows for the v2 gateway and tailer fields, a limits.md
row (signed at read time, so an edit before the read isn't caught; the
gateway signs off the wire), and the TIPS sources paragraph.

Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
…eway; gateway and tailer count once (review of #871)

- A gateway's record of a turn without a tool call joins its transcript
  message by its counts even where the tailer signed it too, so the
  turn counts once; where both signed it, the gateway's counts (signed
  off the wire) are kept. The count pass compares with the counts the
  tailer signed.
- A record joins a transcript turn by message id only from a
  claude-code run whose records name a tool call of that session,
  signed from the turn's first line to MATCH_S after its last; by a
  tool_use id only from an LLM gateway's record (its source). Any other
  record of a transcript's turn is refused: it counts for nothing, the
  transcript's counts stand, and `tracekit tips` and the TIPS page say
  how many. An exchange id counts once in a run, and once across runs
  where no transcript holds it.
- The tailer signs a turn that stopped for a tool use on a tool_use
  line, so the viewer's model line names its calls when a read ends
  before them.
- The viewer's token totals count an exchange id once.
- Docs: the limits rows (made-up lines, read only at the next hook
  event after a kill or park, the 25 MB check, the session's own run
  token), the privacy rows (tool use ids and names, tool_results_sent,
  what output counts reveal), and the TIPS matching paragraph.

Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
…t; the viewer counts a gateway-and-tailer turn once (2nd review of #871)

- In tips' join, a transcript turn takes its records' counts in a pass
  only when they all agree; when they don't, each is refused and the
  turn keeps the counts it had. A run that names one call of the
  session can push a signed turn down to the transcript's counts, but
  no record's counts go over another's. The gateway's counts are still
  kept over the tailer's.
- An exchange id no transcript holds that two runs signed counts once
  when their counts agree, and for neither (refused) when they don't.
- The viewer's TOKENS IN and OUT join a turn the LLM gateway and the
  transcript tailer both signed (by exchange id, tool_use id, or the
  same counts within 5 minutes) and keep the gateway's counts.
- Docs: the limits row says any run that names a call of the session
  can sign for its turns, and that conflicting records drop to the
  transcript's counts; the TIPS paragraph says the same.

Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
… join takes one turn per record (final check of #871)

- A run could hide spend an LLM gateway signed: by signing the
  gateway's exchange ids at other counts (both were refused), or by
  moving a turn in the transcript so the gateway's record fell outside
  its window (refused). A gateway's record now always counts: on a
  conflict only the other records are refused, one that joins no turn
  in its window counts on its own, and two gateways that disagree on a
  turn both count. A run can make tips count more, never less.
- The viewer joins records by their counts one turn per record: two
  turns with the same counts that the gateway and the tailer both
  signed count as two, not three.
- Docs: the limits row and its remedy, and the TIPS paragraph.

Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
The agent's user could write transcript lines under the gateway's exchange
ids (the dev store shows them): pass 1 took the gateway's records for the
tailer's, refused them, and only the made-up turns counted. A gateway's
record now skips the message-id pass, so it joins by tool_use id or counts,
or counts on its own.

Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
@LoopGlitch26
LoopGlitch26 merged commit b0cfac6 into main Nov 15, 2026
35 checks passed
@LoopGlitch26
LoopGlitch26 deleted the agent/issue-865 branch November 15, 2026 12:34
LoopGlitch26 added a commit that referenced this pull request Nov 15, 2026
…the lessons part

Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
This was referenced Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Sign each model turn's usage as it happens, not read later from the transcript

1 participant