Repository navigation
Conversation
added 2 commits
October 10, 2026 08:59
Every commit files the conversation's SSD write on the idle lane, which runs only in a quiet window. Two clients taking turns on one server leave none: one client's request is always queued behind the other's. When a request does not fit, the admission's last step releases the idle conversation and cancelled that queued write, so the conversation existed only in RAM and was lost. Its next turn re-read everything. A field log of two coding agents on a 27B: seven idle sessions of 7.5 to 9.9 GB released with on_ssd_entries 0, each followed by a re-read of 80K to 103K tokens (4 to 6 minutes) that the other agent then waited behind. - SessionBank.release_sessions(write_out_before_release=True) writes each released entry whose own SSD write is still queued (not published, not a lease) to the cold tier first, on the model-owner thread, while the release holds the session. The write streams through spill_entry whatever the entry's size: one tensor's bytes at a time, so it adds little to the memory being given back. The queued job is cancelled afterwards as before, which frees its arrays. - The write is bounded by RELEASE_WRITE_OUT_MAX_S (30 s for the whole release). A write that runs out of time, that the tier refuses (hourly budget, disk room, size cap) or that fails leaves the old behaviour: the entry is dropped. - spill_entry takes give_up_at_s: a caller that is itself the foreground work passes a deadline, which replaces the foreground signal (that signal is up during any request, so the idle-lane yield would stop the write at its first tensor). - Only the admission's idle-conversation step opts in. The shed before a prefill abort, the request's own siblings, the pressure trim and the allocation-failure shed keep cancelling: they run when memory is shortest, or off the model-owner thread. - The release receipt and the guard's idle_release carry written_out_entries and write_out_s; the eviction log records each write's outcome and time (release_write_out). With the SSD cache off, or a tier without the deadline, nothing changes. Tests: tests/test_release_writes_out_unwritten.py (10 tests, real SessionBank and SessionBankColdTier, the engine busy).
The memory guard's pressure trim runs off the model-owner thread. At WARNING it waits up to 60 s for an idle engine, then trims while the request runs. It took an idle conversation whose SSD write was still queued and cancelled that write: the write waits for a quiet window, two clients taking turns never leave one, so RAM held the only copy. A 64K conversation was dropped this way during the other client's cold prefill (the allocator at 0.997 of its limit), and its next turn re-read all 63,881 tokens (173 s). - SessionBank.shrink_to_bytes(protect_unwritten=True) never takes an entry whose own SSD write is still queued (_awaits_queued_write: not published, not a lease). It takes entries already on SSD, or that nothing will write, as before; with only unwritten ones left it stops above its target. - The pressure trim passes it at WARNING only. Such a conversation leaves RAM later: its write runs in the next quiet window and a later trim takes it, or a request that does not fit writes it out before its admission releases it. CRITICAL keeps taking everything idle. - No write from the trim itself: it runs on the asyncio thread while a request may be using the GPU, and at CRITICAL memory is shortest. - The trim's receipt carries bank_unwritten_kept and bank_unwritten_dropped (receipt_out), so a log shows whether a trim dropped a conversation that was in RAM only. Tests: tests/test_pressure_trim_keeps_unwritten.py (5 tests, the real loop, SessionBank and SessionBankColdTier, the engine busy). The loop tests' stand-in banks take the two new keywords, as the real bank does.
jvmenen
pushed a commit
to jvmenen/MTPLX
that referenced
this pull request
Oct 10, 2026
…t section 11, VONDSTEN 128 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
When two clients take turns on one server, an idle conversation's SSD copy is often not written yet: its write waits on the idle lane for a quiet window that never comes. The memory guard's pressure trim takes idle conversations when memory runs high. At WARNING it waits up to 60 s for an idle engine and then trims anyway, while the other client's request runs. Taking a conversation cancels its queued write, so a conversation that existed only in RAM is lost, and its next turn re-reads the whole prompt. In one run a 64K-token conversation was dropped this way during the other client's prefill, and its next turn took 173 s instead of a few seconds.
How it works, in plain terms
At WARNING the trim now passes over a conversation whose own SSD write is still queued. It takes conversations that are already on disk, or that nothing will ever write, as before. A conversation it passes over leaves RAM later in one of two ways: its write runs in the next quiet window and a later trim can take it, or a request that does not fit writes it to disk before releasing it (#618). The trim itself never writes: it runs off the model-owner thread, possibly while a prefill is using the GPU. CRITICAL is unchanged and still takes everything idle, because there memory comes first. When only unwritten conversations are idle, a WARNING trim takes nothing; the trim's receipt reports this.
What changes
SessionBank.shrink_to_bytes(protect_unwritten=..., receipt_out=...); only the pressure trim at WARNING setsprotect_unwritten.pressure_trimreceipt gainsbank_unwritten_kept(entries passed over) andbank_unwritten_dropped(entries taken with their queued write cancelled, at CRITICAL).Stacked on #618 (it uses
_awaits_queued_write): this PR contains #618's commit until that one is merged.Results
Checked for correctness on a 27B with segmented KV, using the #618 scenario at a memory limit where the trim fires during a cold prefill (allocator at 0.998 of the limit, WARNING, after the 60 s deferral). The trim passed over the unwritten idle conversation (
bank_unwritten_kept 1,bank_unwritten_dropped 0). That conversation's next turn restored from RAM (62,336 of 63,881 tokens cached) instead of re-reading all of them. Later, an admission that needed the room wrote the other conversation to SSD before releasing it, and that conversation's next turn restored from SSD. No request was refused.Tests
tests/test_pressure_trim_keeps_unwritten.py(5 tests, the real loop,SessionBankandSessionBankColdTier, the engine busy): WARNING keeps the unwritten conversation and its queued write; WARNING takes a conversation already on disk instead of an older unwritten one; once the write has run, a later trim takes it; without an SSD cache the entry is taken as before; CRITICAL still takes it and counts it. All 5 fail without the change. The pressure-loop tests' stand-in banks accept the new keywords. Full suite passes.🤖 Generated with Claude Code