llama : experimental mlock pinning of hot MoE expert slices (RFC)

[aryan0078]

Overview

RFC / experimental. Adds env-gated mlock() pinning of the statistically
hottest MoE expert slices inside the existing model mmap.

When system RAM is smaller than the routed-expert pool, page-cache LRU
thrashes: every token streams n_expert_used slices per layer and evicts
pages the next token needs. Pinning the hottest (layer, expert) slices makes
them ineligible for eviction, so only cold experts stream from disk.

• LLAMA_EXPERT_PIN_PROFILE — csv of layer,expert[,count], hottest first
  (producible with the moe-trace example, PR examples : add moe-trace to log MoE expert routing decisions #28544)
• LLAMA_EXPERT_PIN_MB — pin budget (default 8192)

Off by default, Linux-only, no behaviour change when the env var is unset.
Slices are addressed as t->data + expert * t->nb[2] on the host buffer;
device-resident expert tensors are skipped.

Honest status

The mechanism is verified (kernel reports the pinned bytes as Locked in
/proc/<pid>/smaps; partial pinning degrades gracefully when
RLIMIT_MEMLOCK is low), but I do not have a benchmark showing an
end-to-end win: the model this was built for (Qwen3.8-Flash-Next) turned out
to route near-uniformly (98.6% of slots active in ~1.3k tokens), where no
pinning policy can beat plain LRU — that finding is documented in a comment
at the pin site. On a skewed router with RAM < expert pool the expected gain
is real but currently theoretical.

Opening as a draft to ask: (a) is this direction acceptable at all, and
(b) if so, should the env vars be promoted to llama_model_params / CLI args?
Happy to close if maintainers prefer to wait for a demonstrated win.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES — the code and this description were prepared with
  an AI assistant (Claude Code). I reviewed the diff and verified pinning
  behaviour on my machine (i7-12700, 30 GiB RAM, RX 6700).

🤖 Generated with Claude Code

[ggml-gh-bot]

Hi @aryan0078, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• Multiple open PRs from a new contributor: We limit new contributors (those without a previously merged PR) to 1 open PR at a time. You currently have 3 open PRs.

• AI-generated content: While code is allowed to be generated by AI, please write the PR description and commit messages on your own without the help of AI.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.