Skip to content

perf(server): warm the usage scan cache after startup - #29731

Open
juliusmarminge wants to merge 2 commits into
t3/usage-parse-off-mainfrom
t3/usage-warm-scan-cache
Open

juliusmarminge wants to merge 2 commits into
t3/usage-parse-off-mainfrom
t3/usage-warm-scan-cache

Conversation

@juliusmarminge

Copy link
Copy Markdown
Member

Stacked on https://git.cafe/pingdotgg/t3code/pulls/29724.

After a restart with a cold or stale scan cache, the first server.getUsageSummary paid for parsing every changed transcript while the user waited: p50 8.6s and p95 34s on a fresh install. The per-file scan cache is persisted, so this work only has to happen once per new or changed file, and it can happen before anyone opens the usage page.

Change

  • UsageService.warmScanCache resolves the transcript directories and parses the default 30-day window (31 days, to cover the client's local-midnight start) into the scan cache, then persists it. It only covers transcript directories; account-based scan readers (Cursor, etc.) still run on request because they make network calls.
  • UsageService.layerWarmup forks it with forkParked, so it starts only after server activation and never delays readiness. It is wired in next to the other post-activation background layers in server.ts. Parsing runs in the layer-1 child processes, so the server's main thread stays mostly idle.
  • A summary that arrives during the warm-up waits for it (a new UsageService.awaitWarmup span) and then reads the warmed cache, instead of parsing the same files a second time. The warm-up skips itself if a summary is already in flight, because that summary warms the cache anyway. Later restarts only parse files that are new or grew.

Before / after

On cups, against the real ~/.claude + ~/.codex data (3,779 files, 292k records, 30-day window):

  • before: the first summary after a cold start parses everything: 7.8–10.9s wall with layer 1, 10.3–12.7s without it.
  • after: that parse runs in the background right after activation. Once it finishes, a summary does the file walk and stat (175–300 ms for 3,802 files), cache lookups, and aggregation, and parses only files that changed since the warm-up.

Tests

A new UsageService.test.ts case:

  • builds the warm-up layer while activation is still pending. The build returns, so startup does not wait, and nothing has been parsed yet.
  • completes activation and gates the warm-up's parse on a Deferred.
  • starts a summary and races "summary waits on the warm-up" (observed via the span) against "summary starts its own parse". It asserts the file was parsed only once, then that the summary totals are correct. Without the wait, the test fails with two parses of the same file.

No sleeps or polling.

Scope and approval: maintainer-approved direction from the 2026-11-02 fleet trace audit

🤖 Generated with Claude Code

@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:M 30-99 changed lines (additions + deletions). labels Nov 2, 2026
macroscopeapp[bot]
macroscopeapp Bot previously approved these changes Nov 2, 2026
@macroscopeapp

macroscopeapp Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Approved at 6806f5b

Macroscope's review found this PR approvable — Warms the existing usage cache after server activation and shares that work with concurrent summaries. The change reuses existing parsing and persistence paths without changing usage defaults or source transcripts.

No code changes detected at b581f66. Prior analysis still applies.

You can add or adjust custom eligibility rules. Learn more.

@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

ℹ️ No successful main baseline artifact is available yet. This run establishes the initial measurement.

Provider Metric Main baseline This PR Impact PR ceiling
Codex Total thread wire — 5.0 KiB — 6.8 KiB ✅
Codex Thread snapshot wire — 3.8 KiB — 4.9 KiB ✅
Codex Live turn WebSocket wire — 1.2 KiB — 2.0 KiB ✅
Codex Live turn WebSocket decoded — 21.6 KiB — 24.4 KiB ✅
Codex Live turn messages — 2 — 8 ✅
Claude Total thread wire — 5.0 KiB — 6.8 KiB ✅
Claude Thread snapshot wire — 3.8 KiB — 4.9 KiB ✅
Claude Live turn WebSocket wire — 1.3 KiB — 2.0 KiB ✅
Claude Live turn WebSocket decoded — 21.9 KiB — 24.4 KiB ✅
Claude Live turn messages — 2 — 8 ✅

Baseline: unavailable · PR result: b581f66 · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 108.5 KiB
  • Claude decoded thread snapshot: 108.8 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

@gitcafe-production
gitcafe-production Bot force-pushed the t3/usage-warm-scan-cache branch from 57bea92 to b7aa3ff Compare November 2, 2026 23:58
@macroscopeapp
macroscopeapp Bot dismissed their stale review November 2, 2026 23:58

Dismissing prior approval to re-evaluate b7aa3ff

macroscopeapp[bot]
macroscopeapp Bot previously approved these changes Nov 3, 2026
@coderabbitai

coderabbitai Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

Recent review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between b7aa3ff and b581f66.

Files selected for processing: 2
  • apps/server/src/usage/UsageService.test.ts
  • apps/server/src/usage/UsageService.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 4 remain after this review.

Walkthrough

UsageService now warms its scan cache with transcripts from the prior 31 days after server activation. Summary scans coordinate with warm-up, and rate loading starts while a summary waits. Tests cover activation gating, shared parsing, and scanning older transcripts omitted by a shorter summary window.

Changes: Usage cache warm-up

Layer / File(s) Summary
Cache scanning and coordination
apps/server/src/usage/UsageService.ts
UsageService adds a 31-day cache warm-up scan and shared completion state. Summary scans wait for active warm-up, while rate loading can proceed during that wait. Warm-up loads the scan cache and settings, scans transcript directories, schedules persistence, and logs failures.
Activation and server wiring
apps/server/src/usage/UsageService.ts, apps/server/src/server.ts, apps/server/src/usage/UsageService.test.ts
The server application includes the warm-up layer. The layer starts warm-up after activation without blocking its launch. Tests cover activation gating, shared parsing, and scanning an older transcript omitted by a one-day summary.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Other

Sequence Diagram(s)

sequenceDiagram
  participant ServerActivation
  participant layerWarmup
  participant UsageService
  participant scanSummary
  ServerActivation->>layerWarmup: releases activation
  layerWarmup->>UsageService: starts warmScanCache
  scanSummary->>UsageService: waits while warm-up is active
  UsageService->>scanSummary: signals warm-up completion
Loading

Possibly related issues

Possibly related PRs

  • pingdotgg/t3code#21776 — This PR adds the warm-up layer to the startup activation wiring introduced there.
  • pingdotgg/t3code#28111 — This PR warms the scan cache using the 31-day window and settings used by the retention behavior changed there.
  • pingdotgg/t3code#29724 — This PR starts cache warm-up through UsageService and exercises the parser-backed scan implementation added there.

Suggested reviewers: maria-rcks, t3dotgg, pixpmusic

Merge Risk: Minimal · up to b581f

The startup cache warm-up remains active, and the previously reported delays have been addressed. No actionable merge-blocking risk remains after normal checks.

Pre-merge checks: Passed 4
Check name Status Explanation
Title check Passed The title clearly and concisely describes the main change: warming the server usage scan cache after startup.
Description check Passed The description explains the problem, implementation, scope approval, performance impact, and focused tests. It uses a "Tests" heading instead of the required "Verification" heading, but the required …
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Description check

Explanation

The description explains the problem, implementation, scope approval, performance impact, and focused tests. It uses a "Tests" heading instead of the required "Verification" heading, but the required verification details are present.

Finishing Touches
Generate docstrings
  • Commit to this branch
  • Create a new PR
Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

CodeRabbit Verify

  • Run CodeRabbit Verify to test the changes in this pull request.

@coderabbitai coderabbitai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Fix comments: Fix CodeRabbit comments on this PR

Comment thread apps/server/src/usage/UsageService.ts Resolved
Comment thread apps/server/src/usage/UsageService.ts Resolved
@macroscopeapp
macroscopeapp Bot dismissed their stale review November 3, 2026 00:11

Dismissing prior approval to re-evaluate 6806f5b

juliusmarminge and others added 2 commits November 2, 2026 16:14
The first usage summary after a restart on a fresh or stale cache paid for
parsing every changed transcript. After activation the server now parses the
default 30-day window into the persisted scan cache in the background; a
summary arriving meanwhile waits for that warm-up instead of parsing the same
files again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ces while waiting

A summary in flight at activation used to make the warm-up skip for the
server's lifetime, even when it covered only an hour or a day. The warm-up
now waits for in-flight summaries and then warms the rest of the window
from cache. A summary that waits on the warm-up now fetches prices in the
meantime instead of afterwards.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@gitcafe-production
gitcafe-production Bot force-pushed the t3/usage-warm-scan-cache branch from 6806f5b to b581f66 Compare November 3, 2026 00:15

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:M 30-99 changed lines (additions + deletions). vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant