ssd: enforce streaming cache floor at one-prefill minimum - #725
Open
bestbug456 wants to merge 1 commit into
Open
ssd: enforce streaming cache floor at one-prefill minimum#725bestbug456 wants to merge 1 commit into
bestbug456 wants to merge 1 commit into
Conversation
Enforce a floor: the cache must hold at least one full prefill's routed working set (n_layers * DS4_N_EXPERT_USED experts). Below that the cache is smaller than a single token's routed set and every slot evicts what it is about to reuse, so prefill always fails with 'cannot reserve a slot'.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hello! While I was try to make ds4 work on my machine I hit some error that where not clear (at least, to me) therefore after a little bit of digging with AI supported I realize that I was miss configuring the system, is not always easy to drive through this type of situation, therefore I would like to propose a change to guarantee that if miss configuration happen the system try to "auto heal" reporting a warning to the user. That been say, here is the technical description of the changes:
Summary
Enforce a floor on the SSD streaming expert cache budget: it must hold at least one full prefill's routed working set (n_layers x DS4_N_EXPERT_USED experts).
Problem
When the cache budget is smaller than a single token's routed expert set, every slot evicts what it is about to reuse. This causes prefill to always fail with "cannot reserve a slot".
Fix
Added a floor check after budget calculation. If the budget is below `DS4_N_LAYER * DS4_N_EXPERT_USED`, it is raised to that minimum and a diagnostic message is printed.
Files Changed
Testing