Skip to content

feat: add ds_load_balance P/D routing with Mooncake prefix hints. - #96

Draft
PixelFlat wants to merge 2 commits into
xLLM-AI:mainfrom
PixelFlat:codex/prefill-time-load-balance
Draft

PixelFlat wants to merge 2 commits into
xLLM-AI:mainfrom
PixelFlat:codex/prefill-time-load-balance

Conversation

@PixelFlat

@PixelFlat PixelFlat commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Add the opt-in ds_load_balance policy to route P/D requests using estimated outstanding prefill compute and decode request/KV load. The name credits DeepSeek's deployment article; this implementation adapts its instance-level routing guidance to the existing Service protocol.

  • Query Mooncake for a reusable contiguous prefix, requiring every configured cache component and TP shard to exist. Reuse TimePredictor to estimate the remaining prefill computation.
  • Choose the healthy P with the smallest sum of estimates for unfinished assigned requests, then reserve the new request's estimate. Equal loads rotate; the new request's estimate is not part of the current ranking.
  • Choose D using a DualPath-inspired token threshold: balance request counts within the threshold, otherwise prefer lower assigned KV length.
  • Bind engine incarnations and reserve load atomically. Release reservations on handoff, completion and cancellation, and account for actual streaming/non-streaming generated-token counts.
  • Add a pinned third_party/Mooncake metadata-only adapter behind USE_MOONCAKE_PREFIX_LOOKUP=ON. The default build keeps it disabled; no vcpkg dependency changes. The Service does not allocate a Store DRAM segment or move KV payloads.

Configuration and startup examples are in docs/ds_load_balance.md.

Validation

  • Mooncake-enabled build and 27 focused policy, predictor, prefix lookup, metrics and admission tests passed. The policy was subsequently renamed from LEAST_TIME to ds_load_balance, including its CLI value, class/files, log tags and tests, without changing the routing rules. The focused suite was rebuilt and rerun after the rename.
  • An isolated real Mooncake Master exercised metadata hits, missing keys and bounded RPC failure handling.
  • Single-host Ascend A3 validation with Qwen3-8B BF16, four NPUs, two TP1 P engines and two TP1 D engines, Python model implementation, ACL graph and task pipeline. Paired xLLM engine revision: 28e32d72a22570edd046c445839ed488e53ae1a0.
  • Measured uncached profiles at 128, 512 and 1536 tokens per P for functional calibration.
Case Result
Multi-turn cross-P reuse First prompt: 1905 tokens; D output: 512 tokens. The client appended the returned text to the next-turn history, without supplying original token IDs. The second turn selected the other P; the router hint and engine reuse both reported 2304 tokens, including 399 tokens from the previous D output.
Four concurrent requests All completed; both P and both D engines participated, with reservations visible to later admissions.
Cancellation One stream was deliberately cancelled; four subsequent probes completed and P/D load counters returned to zero.
Store disk offload Enabled a 2 GiB DRAM segment and 2 GiB bucket-backend disk quota; Store metrics and cache files confirmed disk writes. Cold SSD reload was not exercised in this run.

Ten ds_load_balance requests completed in addition to the deliberately cancelled stream. The multi-turn second request had approximately 669 ms client TTFT. The first request was non-streaming, so its full-response latency is not TTFT. Owned test services were stopped and cleanup was verified.

Scope and limitations

  • These are functional checks, not a large-concurrency benchmark or a performance guarantee. No Mooncake-disabled comparison was run, so the TTFT observation does not establish a speedup.
  • The ledger is local to one Service process. Running prefill requests retain their full reservation until handoff; batching, elapsed execution time, Store/SSD promotion and Host/NPU transfer costs are not fitted.
  • Exact Store key prefixes and compatible cache layouts must be configured explicitly. Metadata hints do not reserve KV and engine GET remains authoritative. No HBM/Host prefix-affinity routing, per-DP-rank routing, dynamic buckets or time-budget chunk sizing is added.
  • Before selecting ds_load_balance, an initial PD engine request with min_tokens failed task-pipeline sampling validation on D. Final cases used max_tokens plus ignore_eos, omitting min_tokens; this separate engine issue is not fixed here. A host-specific CPUinfo read stall also required a private snapshot mount during model initialization.

@PixelFlat PixelFlat changed the title feat: add time-based P/D load balancing with Mooncake prefix hints feat: add ds_load_balance P/D routing with Mooncake prefix hints Oct 23, 2026
@PixelFlat PixelFlat changed the title feat: add ds_load_balance P/D routing with Mooncake prefix hints feat: add ds_load_balance P/D routing with Mooncake prefix hints. Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant