MoE streaming: prefetch off the decode path, time split measured (pin llama.cpp-vulkan a29be7f)

[bong-water-water-bong]

• Pins third_party/llama.cpp-vulkan to a29be7f (fork PR README thank-you and a complete third-party NOTICE, ahead of going public #20). The gate-ahead prefetch is now scored on a worker thread, ONEBIT_MOE_PREFETCH=N sets how many layers ahead it looks (default 1), and ONEBIT_MOE_STATS=1 prints where decode time goes. The fork PR also fixes its CI Test step (Zamba/Zamba2/BlackMamba in test-llama-archs, and the saver for Zamba v1).
• docs/moe-streaming.md gets a new section, "Where streamed decode spends its time":
  • Coder-30B at 4608 slots: prefetch went from 28 tok/s (the old inline scoring) to 43.5-44.8 tok/s, against 42 with prefetch off.
  • The host round trip per layer caps streamed decode at about 57 tok/s.
  • At 1536 slots the drive is the limit.
  • Next item 1 is rewritten.
• Perplexity with prefetch 1 and 2: 21.1029, identical to all-Vulkan.

🤖 Generated with Claude Code

[context7]

Docs7 for 1bit-monster/engine

Result	Status	Action
Deployment	➖ Not used	—
Content review	➖ Did not run. This site has no agent runs available this month. Wait for the monthly reset or check your Docs7 plan.	—

Commit 4441a6b

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit 4441a6b)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 20 - Partially compliant Compliant requirements: README thank you section added NOTICE file updated with complete third-party attribution All copyright lines in NOTICE come from project's own LICENSE or source headers Vendored Lemonade attributed to AMD with changes noted Every pinned submodule attributed Every library fetched by CMake attributed at pinned version Compound licenses stated exactly AMD's NPU firmware noted as not redistributed Non-compliant requirements: None Requires further human verification: Verification that all attribution details in NOTICE match the actual source code and licenses Confirmation that the README thank you section accurately reflects the contributions

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪

🧪 No relevant tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Performance Claim Verification The PR claims that prefetch moved off the decode path improved performance from 28 tok/s to 43.5-44.8 tok/s at 4,608 slots. This performance improvement must be verified with actual measurements on the target hardware (AMD Strix Halo's XDNA 2 NPU) to ensure the claim is accurate and not overstated. | streamed, 1,536 slots (25%) | 8-10 | Through `1bit serve` (a 128-token chat answer, three requests): 18.9, then 28.3 and 27.1 tok/s at 4,608 slots; 7.1, 6.6 and 6.0 at 1,536. The answers match the resident model's. With the prefetch moved off the decode path (below), 4,608 slots gives 17.1, then 38.9 and 39.0. Hardware Limitation Clarification The PR states that "The host round trip caps streamed decode at about 57 tok/s" and that "At 1,536 slots the drive is the limit." These claims need to be validated with actual hardware measurements to confirm they accurately reflect the performance characteristics of the system. - **The round trip caps streamed decode.** 0.37 ms between remaps x 48 layers is 17.6 ms per token, about 57 tok/s, against 12.5 ms (80 tok/s) resident. Each layer's routed ids go to the host and its slot indices come back before the expert matmuls can run. - **Prefetch now pays a little.** Its router scoring first ran inside the remap op, scalar, at about 0.15 ms per layer: prefetch cost 4,608 slots a third of its speed (28 against 42 tok/s). A worker thread now scores the next `ONEBIT_MOE_PREFETCH` layers (default 1) off the decode path. - **At 1,536 slots the drive is the limit.** About 88 misses x 2.1 MB per token is 185 MB, and at 10 tok/s that is about 2 GB/s through `O_DIRECT` on a drive shared with other tenants. More reads in flight (`ONEBIT_MOE_IO` 4 to 32) didn't help, and prefetch trades demand misses for wasted reads (80 against 61 GiB over the run). What helps here is reading fewer bytes: a better hit rate (item 2 below) or smaller experts.

[github-actions]

Persistent review updated to latest commit 4441a6b