Skip to content

bugfix: initialize service tokenizers before serving requests. - #92

Merged
XuZhang99 merged 1 commit into
xLLM-AI:mainfrom
PixelFlat:fix/tokenizer-worker-warmup
Oct 20, 2026
Merged

XuZhang99 merged 1 commit into
xLLM-AI:mainfrom
PixelFlat:fix/tokenizer-worker-warmup

Conversation

@PixelFlat

@PixelFlat PixelFlat commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Description

Requests arriving on a previously unused brpc worker currently clone and load a tokenizer on that request's critical path. An instrumented native-service PD run traced five TTFT outliers to this lazy clone, which took 205–252 ms each. Sending ordinary warmup requests does not guarantee that every future worker has initialized its thread-local tokenizer.

Replace per-thread lazy cloning with a Scheduler-owned tokenizer pool, created before serving requests. Its capacity uses the existing num_threads setting (default 32) and adds no worker threads. Each instance encodes a warmup string, then decodes those IDs with skip_special_tokens=false and true before requests are accepted. Requests borrow an instance exclusively and return it after tokenization; when the pool is exhausted, they wait for an existing instance instead of creating another. This moves initialization cost to startup and keeps tokenizer state isolated between Scheduler/model instances.

Also release the Rust-owned encode buffer after copying token IDs into C++, using the existing tokenizers_free_encode_results export.

Validation

  • Current Linux ARM64 service build and 79 focused C++ tests passed. Four pool regressions cover startup initialization in both directions, exclusive reuse under contention, release after a failed encode, and isolation between models. Changed C++ files passed clang-format 20.1.6 and pre-commit checks.
  • Earlier acceptance of this fix with Qwen3-8B BF16, native-service 1P1D, TP=1/DP=1 and concurrency=1 passed 18 protocol/structure checks, including streaming/non-streaming requests, tools and reasoning.
  • In that acceptance run, all 120 measured requests completed and none exceeded 100 ms TTFT; 20 warmup requests were excluded. The workload used 40/701 input tokens, 64 output tokens and repeated warm prefixes. Post-fix TTFT P95 was 20.85/23.88 ms for the short/long prompts.
  • The earlier diagnosis had extra tracing, so its five outliers establish the initialization cause rather than a strictly matched median-performance A/B comparison. The 120-request result is earlier live-model evidence; this PR preparation reran the build and focused C++ tests.

Summary by CodeRabbit

  • New Features

    • Added pooled tokenizer support for concurrent prompt encoding.
    • Tokenizer instances are initialized and warmed up before use, improving readiness during request processing.
    • Encoding and decoding now support safe concurrent access while preserving configurable special-token handling.
  • Bug Fixes

    • Improved tokenizer resource cleanup after encoding operations.
    • Added validation to prevent scheduler startup with an invalid thread count.
  • Tests

    • Added coverage for tokenizer warmup, concurrent access, reuse, failure handling, and isolation between pools.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-20T08:29:12.913522Z b74295f PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@coderabbitai

coderabbitai Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 9ff433fe-566c-4e4d-9793-574430ef4928
📥 Commits

Reviewing files that changed from the base of the PR and between e31f482 and b74295f.

📒 Files selected for processing: 10
  • tests/xllm_service/tokenizer/CMakeLists.txt
  • tests/xllm_service/tokenizer/tokenizer_pool_test.cpp
  • xllm_service/scheduler/scheduler.cpp
  • xllm_service/scheduler/scheduler.h
  • xllm_service/tokenizer/CMakeLists.txt
  • xllm_service/tokenizer/fast_tokenizer.cpp
  • xllm_service/tokenizer/tokenizer.h
  • xllm_service/tokenizer/tokenizer_pool.cpp
  • xllm_service/tokenizer/tokenizer_pool.h
  • xllm_service/tokenizer/tokenizers/tokenizers.h

Included review availability: This review used your included allowance. Your plan provides up to 2 included reviews per hour; 0 remain after this review.

📝 Walkthrough

The tokenizer package now provides a warmed, fixed-size tokenizer pool with exclusive leases. The scheduler uses the pool for prompt encoding. New tests cover initialization, concurrent reuse, failed-encode recovery, and isolation between pools.

Changes: Tokenizer Pool

Layer / File(s) Summary
Tokenizer pool lifecycle
xllm_service/tokenizer/tokenizer_pool.h, xllm_service/tokenizer/tokenizer_pool.cpp, xllm_service/tokenizer/tokenizer.h, xllm_service/tokenizer/tokenizers/tokenizers.h, xllm_service/tokenizer/fast_tokenizer.cpp, xllm_service/tokenizer/CMakeLists.txt, tests/xllm_service/tokenizer/tokenizer_pool_test.cpp, tests/xllm_service/tokenizer/CMakeLists.txt
The new pool validates its configuration, warms tokenizer instances, and returns leased instances after encode and decode operations. Tokenizer::warmup() exercises encoding and both decode modes. FastTokenizer::encode releases non-empty results. Tests cover warmup, concurrent reuse, failed-encode recovery, and pool isolation.
Scheduler pool integration
xllm_service/scheduler/scheduler.h, xllm_service/scheduler/scheduler.cpp
The scheduler creates a pool sized to its configured thread count and uses it for prompt encoding. The thread-local tokenizer helper is removed.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix

Sequence Diagram(s)

sequenceDiagram
  participant Scheduler
  participant TokenizerPool
  participant Tokenizer
  Scheduler->>TokenizerPool: Encode prompt
  TokenizerPool->>Tokenizer: Lease instance and encode
  Tokenizer-->>TokenizerPool: Return encoded IDs
  TokenizerPool-->>Scheduler: Return encoded IDs
Loading

Suggested reviewers: phantomlei3

Merge Risk: ⚪ Minimal · up to b7429

No actionable merge-blocking risk is established for the tokenizer-pool change. It is mergeable after normal checks.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 5.88% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 34 functions across 8 files. (2 skipped: 2… Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly and concisely describes the main change: initializing service tokenizers before requests are served. It matches the tokenizer pool and startup warmup changes.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

Docstring coverage is 5.88% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 34 functions across 8 files. (2 skipped: 2 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

  • Fix all pre-merge checks with AI
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@XuZhang99
XuZhang99 merged commit dc5c822 into xLLM-AI:main Oct 20, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants