Skip to content

Say why a request found nothing to reuse, and stop labelling normal misses as an SSD fault - #631

Open
jvmenen wants to merge 3 commits into
youssofal:mainfrom
jvmenen:fix/cache-miss-reason-labels
Open

jvmenen wants to merge 3 commits into
youssofal:mainfrom
jvmenen:fix/cache-miss-reason-labels

Conversation

@jvmenen

@jvmenen jvmenen commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Problem

The dashboard request log showed SSD_PREFIX_MISS in orange for almost every miss. The cold tier reports that reason whenever it holds nothing reusable, and the session bank copied it over its own reason. So a new conversation, a prompt that shares less than the reuse minimum (512 tokens) with what is stored, and a turn that was evicted from memory all looked like an SSD fault, although the SSD is only the last place searched and nothing failed.

Change

Server: a plain ssd_prefix_miss is now reported as what it meant:

  • new_session: the bank holds nothing.
  • prefix_divergence_at_token: other conversations are stored, but the shared prefix is below the reuse minimum.
  • evicted: this conversation's own earlier turn left memory and is not on the SSD.

Dashboard labels, with the precise reason in the tooltip:

reason label colour
new_session NEW CHAT grey
prefix_divergence_at_token NO SHARED PREFIX grey
evicted EVICTED yellow
ssd_restore_error:*, ssd_missing_*, ssd_prefix_no_recurrent_boundary SSD ERROR orange
ssd_prefix_miss (older servers) NO SSD MATCH grey

Compatibility

No new reason values; all three already exist in CacheMissReason. ssd_prefix_miss is still reported when the memory lane refused a prefix for another reason (the cause is in last_ram_miss_reason), and real SSD failures keep their reasons. The dashboard still reads ssd_prefix_miss from older servers.

Tests

  • Server: a reason per case in tests/test_ram_miss_reason.py (new conversation, no shared prefix, own turn evicted, another conversation not treated as the owner, SSD failure passed through); three existing assertions updated to the new reasons. Ran with test_cache_bank, test_cold_tier_disk_usage_scan, test_prefill_reread and test_session_bank: all pass.
  • Dashboard: bun test (label and colour per reason, rendered row with tooltip), tsc --noEmit and bun run build clean. The built assets are in a separate commit.

Jeroen van Menen added 3 commits October 13, 2026 17:04
The cold tier reports ssd_prefix_miss whenever it holds nothing reusable,
and the session bank copied that over its own reason. A new conversation, a
prompt sharing less than the reuse minimum with what is stored, and a turn
evicted from RAM all read as an SSD miss.

A plain ssd_prefix_miss is now reported as evicted (this conversation's
own earlier turn left RAM and is not on the SSD), new_session, or
prefix_divergence_at_token. Real SSD failures keep their reasons, and
ssd_prefix_miss remains for a RAM-lane refusal with another cause.
The request log showed every miss as the raw reason in orange, so a new
conversation read SSD_PREFIX_MISS. new_session is now NEW CHAT and
prefix_divergence_at_token NO SHARED PREFIX (grey), evicted is EVICTED
(yellow), SSD restore failures are SSD ERROR (orange). The tooltip carries
the precise reason; ssd_prefix_miss from older servers stays readable.
@jvmenen
jvmenen requested a review from youssofal as a code owner October 13, 2026 15:23
jvmenen pushed a commit to jvmenen/MTPLX that referenced this pull request Oct 14, 2026
…incl. single-rounding fix), 134 (youssofal#629), new 135 (fused lse prefill rounds twice)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant