Skip to content

feat: add layered KV event indexing for cache-aware routing. - #93

Merged
liutongxuan merged 3 commits into
xLLM-AI:mainfrom
PixelFlat:feat/static-kv-events
Oct 22, 2026
Merged

liutongxuan merged 3 commits into
xLLM-AI:mainfrom
PixelFlat:feat/static-kv-events

Conversation

@PixelFlat

@PixelFlat PixelFlat commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Replace the heartbeat/etcd KV cache directory with a background ZMQ consumer for xLLM KV events. The service now discovers eligible registered instances, maintains independent Device and Host residency, and uses that metadata for cache-aware routing without network waits in the request path.

  • Add the kv_indexer module with MessagePack decoding, per-instance/DP prefix indexing, bounded buffering, and snapshot recovery after subscription, disconnects, or sequence gaps. Unsynchronized streams contribute no cache benefit. Scheduled recovery is preserved under sustained live traffic, and failed index state is destroyed after releasing its lock.
  • Discover event and replay endpoints from existing instance registrations using configured port offsets. Retire streams when instances leave or change identity, address, or DP layout.
  • Select the P instance by minimizing lambda * (1 - kv_hit_ratio) + (1 - lambda) * normalized_queue_cost. Clamp the hit ratio to [0, 1], weight Host-only hits separately, and use the minimum benefit across all DP ranks because routing cannot pin a DP rank. Preserve D load-based selection.
  • Remove GlobalKVCacheMgr and its KV heartbeat/etcd path while preserving instance registration, health/load reporting, and the deprecated heartbeat wire field.
  • Add the ZeroMQ dependency, discovery configuration example, internal KV query endpoint, and configuration/protocol documentation.

Validation

  • Service build passed in a Linux ARM64 Ascend development environment.
  • All 18 focused index, decoder, discovery, recovery, internal-query, and CAR tests passed, including real TCP ZMQ fixtures.
  • A new sustained-traffic recovery test failed before the fix and passed afterward, covering both initial subscription and recovery after a sequence gap.
  • Full service suite: 181/181 passed, including the filter-name regression updated to use a supported filter and check both missing and present input.
  • GLM-5.3 W8A8 six-layer native RPC 2P2D integration passed with one NPU per engine (TP=1, DP=1): 20 real completion requests, automatic P event discovery, and both D instances exercised.
  • All 8 repeated-prefix probes selected the original P instance and reported 384 cached tokens. After Device eviction, a Host-only prefix still selected that P and restored 384 tokens to Device; index state and engine/client cache-hit counters agreed.
  • clang-format 20.1.6 and git diff --check passed.

Scope and limitations

The consumer follows xLLM's current full-snapshot recovery protocol and assumes one model per service cluster. The integration verifies event indexing, layered cache reuse, and routing correctness; it does not assess reduced-model output quality. Existing engine shutdown behavior still required timeout-based cleanup during the integration run.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-22T03:23:43.994501Z d27f5a3 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d27f5a3eb0

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread xllm_service/kv_indexer/kv_index.cpp Resolved
@PixelFlat PixelFlat changed the title feat: add layered KV event indexing for cache-aware routing feat: add layered KV event indexing for cache-aware routing. Oct 22, 2026
Preserve pending recovery deadlines until a snapshot is requested, retire failed index state after unlocking, and cover initial and post-gap recovery under sustained traffic.
@PixelFlat
PixelFlat force-pushed the feat/static-kv-events branch from 55e53a1 to 15611be Compare October 22, 2026 04:11
Use the supported string filter so the regression exercises context-path normalization without relying on the pinned minja default filter accepting an omitted fallback. Verify defined filter results for both missing and present input.
@liutongxuan
liutongxuan merged commit a1251f4 into xLLM-AI:main Oct 22, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants