Repository navigation
perf(storage): prune debug retention from a single artifact scan - #729
Merged
Merged
Conversation
Refs #728 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kk16MNVJtWAnCeGxGRuHqa
pruneDebugStorage runs every five minutes for the life of the process and used to call collectArtifacts() once per pass — three parses of the multi-MB debug-bundle ledger and three stat walks over every tracked file per prune, on the main thread. The passes now run over one collected list with an in-memory live set; byte accounting subtracts what removeArtifact reports, exactly as the old code did between re-scans, so pass outcomes are unchanged. The pass logic is exported with the remover injected so it is unit-testable in memory. Fixes #728 Refs #103 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kk16MNVJtWAnCeGxGRuHqa
…can trade-off Review follow-ups: the remover now answers whether the artifact is gone and the passes charge the artifact's collected bytes, so accounting cannot drift between passes; the comment states what a single scan trades away (bytes/mtime snapshot, files created or removed mid-prune) and why that is acceptable for the seconds a prune takes; the cap and budget tests are built so the protected and active artifacts are actually reached while still over cap/budget; the artifact type is exported directly. Refs #728 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kk16MNVJtWAnCeGxGRuHqa
Owner
Author
|
Self-review pass (adversarial review of the first cut) — resolved in HEAD:
Out of scope, filed separately: ghost logs are protected in every pass so the "cap handles abandoned ghost files" comment is currently false. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
pruneDebugStorage(src/main/storage/debugRetention.ts) runs every five minutes for the life of the process —performanceLog.tsschedules it on every perf flush. Each run calledcollectArtifacts()three times, once per pass (TTL, per-bucket cap, global budget). Each call re-reads and re-parses the debug-bundle ledger (18.4 MB / 45,428 lines on this machine → ~136kJSON.parsecalls and ~55 MB of transient strings per prune) and stat-walks every tracked file (~6,500statround-trips per prune), all on the main thread. Details in #728.What changed
collectArtifacts()per prune. The three passes now run over that single list through a new exportedrunPrunePasses(artifacts, policy, remove)that keeps an in-memory live set: an artifact removed by an earlier pass is dropped from the set and never re-offered.pruneDebugStorageis now policy resolution + one collect + the passes.DebugStorageArtifact,DebugStoragePrunePolicyand the passes result are exported so the pass logic can be tested without disk.Linked issues
Fixes #728. Refs #103, #372.
Verification
NODE_ENV=test npx vitest run --project unit src/main/storage/debugRetention.test.ts— 8 tests passed: the three existingcollectSessionRecordingDirscontracts plus five new in-memoryrunPrunePassescontracts (TTL removes only stale unprotected artifacts; cap pass trims oldest inactive artifacts of an over-cap bucket only; budget pass trims oldest-first across buckets; an artifact removed by one pass is never offered to a later one; a removal that freed nothing is kept in the working set and not counted).npx tsc -p tsconfig.node.json --noEmit— clean.Not in scope
The proxy bucket exceeding its cap (8.4 GB vs 3.86 GB): live Codex runs sit inside
ACTIVE_GRACE_MSand single runs reach multiple GB because the Responses proxy mirrors every SSE chunk. That needs a per-run capture cap in codex-headless; tracked separately.Plan:
docs/superpowers/plans/2026-09-03-debug-retention-single-scan.md.🤖 Generated with Claude Code
https://claude.ai/code/session_01Kk16MNVJtWAnCeGxGRuHqa