Skip to content

Parse growing Claude logs incrementally and fix usage accuracy edges (SP4 batch B) - #24

Open
donky-ey wants to merge 5 commits into
mainfrom
port/sp4-b-claude-scan
Open

donky-ey wants to merge 5 commits into
mainfrom
port/sp4-b-claude-scan

Conversation

@donky-ey

Copy link
Copy Markdown
Owner

Summary

SP4 batch B: Claude log scan performance and usage accuracy. Branched from main after #20; it relies on #20's isolated support folder for tests.

Benchmark (chattymin#337)

PTB_BENCH=1 swift test --filter ClaudeScanBenchmarkTests, with a 102 MB synthetic session log (117,000 lines, 39,000 usage lines), same harness on both sides:

before (main) after
cold parse 11,376 ms 3,289 ms
refresh after appending one line (the issue's steady state) 11,295 ms 118 ms
unchanged refresh 75 ms 69 ms
peak memory footprint after the cold parse 219 MB 80 MB

Held: chattymin#318 (Claude-reported cost-state)

The record format is confirmed from real Claude Code transcripts on this machine (read-only, structure only):

  • type:"cost-state", with startTime in epoch milliseconds;
  • modelUsage.<model>.{inputTokens, outputTokens, cacheReadInputTokens, cacheCreationInputTokens, costUSD};
  • the per-model costUSD values sum to totalCostUSD.

Its meaning does not match upstream's assumption:

  • The ledger covers only the run since its startTime, not the whole file. 13 of the 22 files with a ledger contain earlier entries (in one, 6,661 of 6,737). Several have an empty modelUsage.
  • Ledger tokens are 1.07–17× the file's own entry tokens. The ledger includes calls the file does not log. 13 of these sessions have subagents, whose transcripts carry no ledger and are parsed and priced separately.

Spreading the last ledger over every entry of the file by tokens, as upstream does, would erase the earlier entries' cost and double-count subagent cost. The fork already prices the Claude 5 family, so the gain is small. I did not port it; the spec and plan record the evidence. This is a decision for the maintainers.

Type of change

  • Bug fix
  • New feature
  • Refactor / cleanup
  • Documentation
  • Other:

UI changes

None.

Verification

  • ./scripts/test-gate.sh:
    • Executed 2036 tests, with 12 tests skipped and 0 failures (0 unexpected)
    • ✓ 게이트 통과 — 로직 코어 라인 커버리지 93.98% >= 75%
    • The first two runs failed only on the known-flaky SessionKeySettingsRenderingTests:46, the scroll timing that batch F's b716054 addresses. The third run was clean, and the test passed 5/5 alone. The machine was under heavy WindowServer and codesign load from other work.
  • New tests (ClaudeIncrementalParseTests, 9):
    • tail-only reads (bytes observed)
    • a streaming turn across the boundary
    • a half-written last line
    • truncation
    • a same-size rewrite
    • a rewritten prefix that grew
    • the incremental result equals a full parse (Review Focus 3)
    • no duplicate accumulation in the stored blob
    • legacy blobs without an offset
  • Mutation checks, each caught by its test:
    • always parsing from 0
    • dropping the fingerprint check
    • dropping the merge dedup (caught by the stored-blob test; the return value is deduped again downstream)
    • counting the half-written line into the offset
  • ISO8601ParserSharingTests: every timestamp format still parses, and 2,000 concurrent calls agree.
  • The fix: hide providers with zero-token synthetic activity chattymin/PokeTokenBar#333 test fails without the filter.

Hotspot files

None (LocalUsageCache, LocalUsageReader, Models (ISO8601Parser only), ModelPricing). UsageStore.swift is not touched.

Not addressed

  • A cold scan that is interrupted still does not persist partial progress. The cache is saved only after all roots are scanned, so a relaunch mid-scan starts over. Separate from the steady-state cost fixed here.
  • Haiku 5.5 has a higher price for prompts over 100k tokens, which the table does not model (it uses the ≤100k row).

Checklist

  • swift build and swift test pass locally
  • PR title and description are written in English
  • UI changes are described above (none)
  • No copyrighted assets, secrets, or private tooling references are committed (see CONTRIBUTING)
  • Tests were added or updated for this change

…issue chattymin#337)

ISO8601Parser built a new ISO8601DateFormatter on every call, on the
assumption that it ran rarely. It runs once per assistant line of every
local session log, so on a large history formatter construction dominated
the cold scan. Keep one fractional and one plain formatter behind a lock,
the pattern LocalAdditionalUsageProvider already uses; parsing behaviour,
including the fractional-width fallback, is unchanged.
…hattymin#337)

The cache keyed blobs on (path, mtime, size), so the active session file,
which changes on every refresh and can reach hundreds of MB, was re-read
from byte 0 each time and three passes per refresh did it repeatedly.

Blobs for Claude now record the offset of the last complete line. When a
file only grew, the cache parses the tail from that offset and merges it
with the cached entries (dedupKeepMax, since a streaming turn repeats its
id across the boundary). A shrink, a same-size change, a missing offset
(older cache files), or a changed fingerprint of the 256 bytes before the
offset (a rewritten prefix) falls back to a full parse. A trailing line
with no newline is parsed but left out of the offset, so it is read again
once complete. Parsing streams 1 MB chunks through the existing line reader
(generalized from the Codex one) instead of loading the file as a String.
The offset lives in usage-cache.json, which the isolated support folder
already covers for tests.

Upstream PR chattymin#338 (unmerged) was used as a reference; the fingerprint, the
chunked reader and the tests are this fork's.

Benchmark (PTB_BENCH=1 ClaudeScanBenchmarkTests, 102 MB synthetic log,
39,000 usage lines), before -> after:
  cold parse              11,376 ms -> 3,289 ms
  refresh after 1 append  11,295 ms ->   118 ms
  unchanged refresh           75 ms ->    69 ms
  peak footprint (cold)      219 MB ->    80 MB

Tests: tail-only reads (bytes observed), a streaming turn across the
boundary, a half-written last line, truncation, same-size rewrite, a
rewritten prefix that grew, incremental result equal to a full parse, no
duplicate accumulation in the stored blob, and legacy blobs without an
offset. Each guard was checked by mutating the code it protects.
…eam chattymin#333)

Claude Code can write <synthetic> assistant records whose usage is all
zero even when no request ran. The store already ignored them for display
(totalTokens > 0), but activeBlock still let them create a 0-token block or
pull the block start earlier, lowering tokens per minute. Filter them out
of the block window. Only the reader half of chattymin#333 is ported; the store half
was already here.
The fork priced Sonnet 5.5 cache reads at $0.20/MTok; upstream changed it to
$0.10. Checked against the primary source on 2026-10-16,
https://platform.claude.com/docs/en/about-claude/pricing: Sonnet 5.5 cache
hits are $0.10/MTok (cache hits on Opus 5.5 and Sonnet 5.5 are 0.05x base
input). Every other Claude row in the table matches that page. Cache blobs
keep raw token counts and are priced at aggregation, so no cache version
bump is needed. Only the pricing part of upstream dc6de41 is ported.
The cost-state format is confirmed from real Claude Code transcripts, but
the ledger covers only the run since its startTime and includes calls the
file does not log (subagent transcripts carry no ledger). Spreading the last
ledger over every entry, as upstream does, would erase earlier entries' cost
and double-count subagents. Hold the port and record the evidence.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants