Repository navigation
Conversation
Adds ggml_expert_cache_entry and ggml_backend_sched_expert_cache structs with find_or_create helper, free/reset lifecycle hooks, and two public API functions (ggml_backend_sched_set_expert_cache, ggml_backend_sched_get_expert_cache_stats) for Task 1 of MoE GPU weight freshness cache infrastructure. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace new/delete with calloc/free for ggml_backend_sched_expert_cache allocation. Fix partial-allocation hazard in expert_cache_find_or_create by allocating both buffers before committing the entry. Remove unused input_cpy parameter from find_or_create. Add forward declaration of ggml_backend_sched_expert_cache before ggml_backend_sched struct. Document reset behavior with a comment explaining entries survive but populated state is wiped. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ration Wire expert_cache through common_params → llama_context_params → llama_cparams, enable ggml_backend_sched_set_expert_cache after sched creation, and log stats on context destruction when the cache was active. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…nt on exit Stats are already gated on hits+misses>0 in the context destructor, so the separate flag added no real value. Removing it reduces surface area and eliminates the forwarding gap between common_params and llama_cparams.
Adds test_expert_cache_hit_on_second_token which builds a minimal MUL_MAT_ID compute graph with CPU weight buffers and a GPU copy target, runs two identical tokens through the scheduler, and asserts that hits and bytes_saved increase on the second token. The test self-skips when no GPU backend is present; full behavioural correctness is covered by the Task 5 perplexity-parity run.
…llel sched loss Two bugs found during code review: 1. bytes_saved double-counted FATE-copied experts: Phase 1 would copy an expert and mark its populated bit, then Phase 2 would see populated=true and count it as n_hits + bytes_saved even though the bytes were copied just moments earlier. Fix: add fate_mask bitset (zeroed before Phase 1, set alongside populated during FATE copies). Phase 2 only counts n_hits/ bytes_saved for experts populated from a PREVIOUS token, not from this call's FATE phase. Phase 1 now also adds to bytes_copied for accounting. 2. Expert cache silently lost on pipeline-parallel fallback: when compute buffer allocation fails and the scheduler is recreated without pipeline parallelism, ggml_backend_sched_set_expert_cache() was not called on the new sched instance. Fix: re-enable expert cache after sched recreation. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…orcing, stats API Upgrade the expert cache from boolean dedup-only to a configurable N-slot LFRU (Least-Frequently/Recently-Used) eviction cache with FATE (Future Anticipated Token Expert) prefetch. Key changes: - --expert-cache-slots N replaces --expert-cache (N=0 off, -1=dedup, N>0=slots) - LFRU eviction with frequency decay (cross-multiply avoids float division) - FATE pre-loads previous token's routed experts for next-token prediction - graph_get_cb forces MUL_MAT_ID to GPU when expert weights are on CPU, enabling the cache to intercept CPU→GPU copies (fixes 0 hit/miss stats) - N-slot tensor for decode (ne[2]=N), full tensor for prefill (dedup path) - Prefill overflow handling with per-expert eviction (no silent corruption) - CUDA MMQ padding (512 bytes) on last GPU slot to prevent NaN reads - Public stats API: llama_expert_cache_stats() / _reset() - llama-bench integration: --expert-cache-slots, stats printed after runs - Write-back skip optimization: avoids redundant GPU ids_tensor writes - Extracted expert_cache_find_evict_slot() helper (DRY eviction logic) Validated: 96.7% hit rate, 1.79x decode speedup on GPT-OSS-120B with RTX PRO 2000 8GB (--n-cpu-moe 36 --expert-cache-slots 16). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Closed
4 tasks done
github-actions
Bot
added
testing
|
Hi @e1n00r, thanks for your contribution! Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:
Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below. |
nick-pape
added a commit
to nick-pape/llama.cpp
that referenced
this pull request
May 15, 2026
Documents the slot-pool + ids-remap pattern (this fork's v2 design, shared with vLLM #37190 / tinyserve / e1n00r ggml-org#21609) and the four portable contributions this fork adds on top: 1. Per-op cudaMallocAsync scratch for prefill overflow 2. CPU intercept floor for high-miss ops 3. GPU-side expert->slot remap via mapping_tensor 4. Empirical LRU win (workload-specific; re-sweep when porting) Framed for re-implementation on top of vLLM PR #37190 once it merges upstream. Includes file refs, commit SHAs, measured results, and a port checklist. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
4 tasks done
simonyos
added a commit
to simonyos/llama.cpp
that referenced
this pull request
Oct 16, 2026
- credit ggml-org#29949: its LRU_FILL kernel already fills the pool by reading pinned memory, so the kernel-copy idea is not ours; what we add is the measurement of the DMA stall on a side stream - remove the unmeasured claim that ggml-org#29949's GPU-fill mode is slower on PCIe 3.0; same-machine numbers are being measured - list earlier llama.cpp work (ggml-org#20757, RFC ggml-org#28248, closed PRs ggml-org#21609, ggml-org#21614, ggml-org#21620, ggml-org#24524) - note that our speed-ups use the best --n-cpu-moe split as baseline, not all experts on the CPU, and that the CPU-miss vs GPU-fill trade-off depends on PCIe speed Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR withdrawn by author.