Skip to content

perf: cache compiled chat templates - #1518

Merged
inureyes merged 2 commits into
mainfrom
perf/issue-1235-chat-template-cache
Aug 30, 2026
Merged

perf: cache compiled chat templates#1518
inureyes merged 2 commits into
mainfrom
perf/issue-1235-chat-template-cache

Conversation

@inureyes

@inureyes inureyes commented Aug 30, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Summary

  • Cache the compiled MiniJinja environment on each ChatTemplateProcessor lifetime so typed, raw, history, and probe renders reuse one compiled template.
  • Keep per-request messages, tools, kwargs, generation-prompt mode, and rejection handling outside the cache so request isolation and fix(server): /v1/responses and the offline CLI still swallow template rejections that /v1/chat/completions now returns 400 for #1176 TemplateRejection/fallback semantics remain unchanged.
  • Add focused cache, byte-identity, clone/concurrency, parse-failure, rejection/fallback, and bounded performance-smoke coverage.
  • Document the cache interaction in the chat-template model support notes.
  • Add committed English and Korean technical reports for this open PR because TECHNICAL_REPORTS/.keep-reports is present.

Validation

  • PASS: cargo fmt
  • PASS: cargo test --lib cached_template
  • PASS: cargo test --lib server::chat_template::tests
  • PASS: cargo test --lib server::reasoning_effort_tests
  • PASS: cargo test --lib history_render
  • PASS: cargo test --lib cached_template_performance_smoke_reports_delta -- --ignored --nocapture (400 debug-profile renders: fresh 27.248578ms, cached 7.99117ms)
  • PASS: cargo clippy --lib --tests -- -D warnings
  • PASS: git diff --check
  • PASS: hosted checks on report commit fe1f5d0: cargo-clippy, cargo-deny, cargo-fmt, OpenXLA feature compile, Detect changes, crate versions, cross-repo refs, kernel dtype keys, license/cla, and llama-compat manifest. MLX pin extraction and OpenXLA feature link skipped by change detection.
  • SKIP: broad cargo workspace tests, serial all-tests, workspace clippy, and cold release builds per watchdog guard.

Reports

  • TECHNICAL_REPORTS/1518-chat-template-cache-20260830.en.md
  • TECHNICAL_REPORTS/1518-chat-template-cache-20260830.ko.md

Closes #1235

@inureyes
perf: cache compiled chat templates
5e202e5 
Cache the MiniJinja environment on ChatTemplateProcessor so typed, raw, history and probe renders reuse the same compiled template across the processor lifetime instead of rebuilding filters and bytecode on each call.

Keep per-request messages, tools and kwargs outside the cache, cache immutable parse failures, and preserve the TemplateRejection sentinel/fallback split added for template-side request rejections.

Refs #1235
@inureyes inureyes added status:review Under review type:performance Performance improvements priority:medium Medium priority area:inference Generation, sampling, decoding (incl. speculative, DRY) labels Aug 30, 2026
@inureyes
docs: add technical report for PR #1518
fe1f5d0 
Add bilingual technical reports for the chat template compile cache implementation in PR #1518.

Refs #1235
@inureyes inureyes added status:done Completed and removed status:review Under review labels Aug 30, 2026
@inureyes
inureyes merged commit 972b5f3 into main Aug 30, 2026
12 checks passed
@inureyes inureyes mentioned this pull request Aug 30, 2026
Merged
inureyes added a commit that referenced this pull request Aug 30, 2026
@inureyes
docs: correct store report rebase references
b885389 
Clarify that the runtime-settings behavior came from PR #1516 and that the later rebase target was main through PR #1518, the chat-template-cache head.

Refs #1248
inureyes added a commit that referenced this pull request Aug 30, 2026
@inureyes
fix(server): bound response stores by bytes (#1519)
46811ff 
## Summary

- Add approximate retained-byte budgets to the in-memory `/v1/responses` response store and conversation transcript store, with checked/saturating accounting and running totals.
- Replace per-eviction linear victim scans with a `BTreeSet` LRU index so live entries keep one bounded metadata node and eviction removes the oldest entry in O(log n).
- Expose `--responses-store-max-bytes` / `MLXCEL_RESPONSES_STORE_MAX_BYTES` and `--conversation-store-max-bytes` / `MLXCEL_CONVERSATION_STORE_MAX_BYTES` on both server entry points, then document the knobs.
- Preserve the runtime-settings work merged by PR #1516 (`01c20350`) while rebasing onto main through PR #1518 (`972b5f3a`, chat-template-cache head); the only conflict was the colocated `mlxcel-server` test block, and both settings tests plus byte-budget tests are retained.

## Validation

- PASS: `cargo test --lib responses_store -- --nocapture` (16 tests; byte-only eviction, count+byte pressure, oversized single entry, exact boundary, replacement, access-order refresh, TTL, zero/extreme budgets, running total)
- PASS: `cargo test --lib conversation_store -- --nocapture` (13 tests; byte-only eviction, count+byte pressure, oversized single transcript, exact boundary, update/replacement, access-order refresh, TTL, zero/extreme budgets, running total)
- PASS: `cargo test --lib store_byte_budgets_round_trip_through_into_startup_config -- --nocapture`
- PASS: `cargo test --bin mlxcel serve_store_byte_budget -- --nocapture`
- PASS: `cargo test --bin mlxcel-server store_byte_budget -- --nocapture`
- PASS: `cargo test --bin mlxcel-server settings_cli_mlxcel_server -- --nocapture` (preserved runtime-settings PR #1516 while rebasing onto main through PR #1518)
- PASS: `cargo test --bin mlxcel settings_cli_mlxcel_serve -- --nocapture` (preserved runtime-settings PR #1516 while rebasing onto main through PR #1518)
- PASS: `cargo test --bin mlxcel settings_cli_build_startup_input_defaults_off_and_propagates_enablement -- --nocapture` (preserved runtime-settings PR #1516 while rebasing onto main through PR #1518)
- PASS: `cargo test --lib settings_cli -- --nocapture` (preserved runtime-settings PR #1516 while rebasing onto main through PR #1518)
- PASS: `cargo test --test llama_compat_manifest manifest_option_claims_hold_on_both_server_binaries -- --nocapture`
- PASS: `python3 scripts/ci/check_llama_compat_manifest.py`
- PASS: `cargo fmt --all --check`
- PASS: `cargo clippy --lib --bin mlxcel --bin mlxcel-server -- -D warnings`
- PASS: `git diff --check`
- PASS: static scan found no conflict markers or old `min_by` / `min_by_key` / `evict_to_capacity` victim-scan code in the touched stores.

## Hosted Checks

- PASS: cargo-clippy, cargo-deny, cargo-fmt, OpenXLA feature compile, Detect changes, crate versions, cross-repo refs, kernel dtype keys, license/cla, llama-compat manifest.
- SKIP: MLX pin extraction and OpenXLA feature link were skipped by workflow conditions.

## Technical Reports

- `TECHNICAL_REPORTS/1519-store-byte-bounds-20260830.en.md`
- `TECHNICAL_REPORTS/1519-store-byte-bounds-20260830.ko.md`

## Limits

- These validations are deterministic unit/config/static checks. I did not run broad workspace tests, cold release builds, real model checkpoints, ffmpeg, or hardware qualification for this server-memory accounting change.

Closes #1248
@inureyes inureyes self-assigned this Aug 31, 2026
@inureyes
inureyes deleted the perf/issue-1235-chat-template-cache branch October 6, 2026 12:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@inureyes inureyes

Labels

area:inference Generation, sampling, decoding (incl. speculative, DRY) priority:medium Medium priority status:done Completed type:performance Performance improvements

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

perf(server): cache the compiled chat template instead of re-parsing it on every render

1 participant

@inureyes