Skip to content

feat: add hard session affinity routing. - #94

Open
PixelFlat wants to merge 4 commits into
xLLM-AI:mainfrom
PixelFlat:feat/session-aware-routing
Open

PixelFlat wants to merge 4 commits into
xLLM-AI:mainfrom
PixelFlat:feat/session-aware-routing

Conversation

@PixelFlat

@PixelFlat PixelFlat commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Add SESSION_AWARE routing using the client-provided X-Session-ID. The first request uses RR or CAR placement; subsequent requests reuse the bound DEFAULT instance or P/D targets while those targets remain eligible.

The policy coordinates concurrent initial requests, confirms bindings after dispatch or successful engine output, protects active requests from expiry and eviction, and starts the idle TTL when the last active request finishes. Instance identity checks and versioned leases prevent stale completions from changing a newer binding. Idle entries use bounded LRU eviction; a full table of active sessions rejects new session placement. Soft affinity is outside this change.

Fallback placement, including CAR prefix lookup, runs outside the session-table lock and reconciles concurrent provisional bindings afterward. Store selected P/D names and incarnation IDs together in Routing, reused directly by requests and session bindings. Session placement requires nonempty incarnation IDs on every selected engine; missing IDs reject the placement instead of using registration timestamps. Ordinary routing without a session header remains available to engines without IDs. Validate identities at binding, admission, metric release, instance cancellation and Channel acquisition; validate and acquire the Channel under the same instance lock. Streaming HTTP errors retain the upstream response body while invalidating affinity independently.

Encapsulate provisional bindings, version guards, idle TTL, LRU and request holds in SessionTable. SessionAwarePolicy only coordinates reuse and fallback placement; SessionLease delegates lifecycle transitions and skips repeated completion notifications. Session bindings store only Routing metadata rather than load snapshots; no separate identity or session-route wrapper is needed.

Read the header on completions, chat completions and Anthropic messages, and connect lease completion to native xLLM results and HTTP backend responses. Preserve the original chat JSON for HTTP forwarding during normalization. Document the routing flags and process-local scope.

Validation completed in a Linux ARM64 development environment:

  • Service build passed.
  • 194/194 service tests passed, including all 13 new session and header tests.
  • Channel lookup regression rejects replaced P/D incarnation IDs at binding, admission and dispatch, then accepts the newly selected route. Re-registration timestamps changing under the same ID preserve affinity.
  • Regression tests cover streaming HTTP 400/503 error-body forwarding with and without a session, missing incarnation IDs on DEFAULT/P/D targets and recovery after IDs become available, and blocked fallback placement with concurrent same-session and unrelated requests.
  • All 13 session/header cases passed 30 consecutive repetitions (390 successful test runs), including P/D failover and stale completion guards.
  • The real service executable with isolated etcd and three observable HTTP mock backends passed 629 engine requests: 8 bursts of 16 concurrent first-turn clients and 20 sessions with 20 requests each showed no target drift. Further checks passed model/header isolation, blank-header RR, same-address incarnation replacement with a late old-stream error, offline-engine reselection, CAR load reversal, missing-ID rejection and recovery, active-stream TTL, failure rebinding, active-capacity protection and idle LRU eviction.
  • Real Ascend NPU validation passed 43 Qwen3-8B BF16 requests on 2P2D, each TP=1/DP=1 with one device per instance. Per-engine prompt and generated-token counters confirmed stable P/D targets across changing prompts, chat/completions, JSON/SSE and three bursts of four concurrent first-turn requests. Missing/blank headers used RR. A 512-token stream retained its binding beyond the configured 2-second idle TTL; after stream completion and idle expiry, the session selected another P/D pair. The service was rebuilt; unchanged engine artifacts were verified against their previous successful-build hashes. All test-owned processes were cleaned up; engine shutdown required timeout SIGKILL.
  • clang-format 20.1.6 and git diff --check passed.

Validation includes CPU mocks and real same-host NPU P/D inference. Failover, admission, cancellation and stale Channel checks were exercised in regression tests. Performance benchmarks were not run. Bindings are local to a service process and select instances rather than internal engine DP ranks.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-23T14:20:56.800678Z 0e1e4b0 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e8e6cc2dda

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread xllm_service/http_service/service.cpp Resolved
@PixelFlat
PixelFlat marked this pull request as draft October 23, 2026 11:34
@PixelFlat
PixelFlat marked this pull request as ready for review October 23, 2026 14:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant