Repository navigation
bugfix: initialize service tokenizers before serving requests. - #92
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info
📝 Walkthrough
Merge Risk: ⚪ Minimal · up to
|
Description
Requests arriving on a previously unused brpc worker currently clone and load a tokenizer on that request's critical path. An instrumented native-service PD run traced five TTFT outliers to this lazy clone, which took 205–252 ms each. Sending ordinary warmup requests does not guarantee that every future worker has initialized its thread-local tokenizer.
Replace per-thread lazy cloning with a Scheduler-owned tokenizer pool, created before serving requests. Its capacity uses the existing
num_threadssetting (default 32) and adds no worker threads. Each instance encodes a warmup string, then decodes those IDs withskip_special_tokens=falseandtruebefore requests are accepted. Requests borrow an instance exclusively and return it after tokenization; when the pool is exhausted, they wait for an existing instance instead of creating another. This moves initialization cost to startup and keeps tokenizer state isolated between Scheduler/model instances.Also release the Rust-owned encode buffer after copying token IDs into C++, using the existing
tokenizers_free_encode_resultsexport.Validation
Summary by CodeRabbit
New Features
Bug Fixes
Tests