Repository navigation
feat(server): add Anthropic-style /v1/messages API endpoint - #74
Merged
Merged
Conversation
Add an Anthropic Messages-compatible API surface to mlxcel-server so clients written against the Anthropic schema can use the server without a translation shim. Ported from upstream mlx-vlm `server/anthropic.py` (PR #1196) plus the tool-result image-handling fix (PR #1200). New endpoints: - `POST /v1/messages` (and `/messages` alias): create a message, synchronous or SSE streaming. - `POST /v1/messages/count_tokens` (and `/messages/count_tokens` alias): prompt token count. The request schema covers the Anthropic surface: top-level `system` (string or text-block array), `messages`, `max_tokens`, `stop_sequences`, `temperature`/`top_p`/`top_k`, function `tools` + `tool_choice`, and `thinking`. Content blocks support `text`, `image` (base64 and url), `tool_use`, and `tool_result`, including image content nested inside `tool_result` blocks so vision models consume tool-produced images. Streaming emits the Anthropic SSE sequence (`message_start`, `content_block_start`/`delta`/`stop`, `message_delta`, `message_stop`, `error`) with `usage` populated on both the non-streaming response and the streaming `message_delta`. The implementation reuses the existing generation, chat-template, media, tool-call, and thinking-budget plumbing rather than duplicating the generation loop: - `types/anthropic_request.rs`, `anthropic_response.rs`, `anthropic_stream.rs`: inbound/outbound Anthropic types and the typed SSE event enum. - `anthropic_translator.rs`: Anthropic request to internal `ChatCompletionRequest` (system/messages/tools/tool_choice, image-in-tool-result), plus outbound content-block assembly, stop-reason mapping, and stop-sequence truncation, mirroring the Responses API translator. - `streaming_anthropic.rs`: the SSE channel and the content-block index state machine. - `routes/anthropic.rs`: thin handlers that call `prepare_chat_request_with_cache`, `build_generate_options`, the model provider, and the shared tool-call helpers. Both the message and count_tokens routes reject oversized tool arrays via the shared `MAX_TOOLS` bound to prevent template-rendering DoS. - Route registration in `app.rs`; module wiring in `types/mod.rs`, `mod.rs`, `routes/mod.rs`. User-facing API documentation is deferred to a follow-up; this change is the functional endpoint plus its unit tests. Verified with cargo fmt, cargo clippy (metal,accelerate, all-targets, -D warnings), and 44 new unit tests covering request translation, content-block assembly, stop-reason mapping, the tools DoS guard, and the streaming state machine.
inureyes
added
type:enhancement
inureyes
added a commit
that referenced
this pull request
May 26, 2026
Bump version to 0.0.31. Highlights: MiniCPM-V 4.6 and RT-DETRv2 model support plus an Anthropic-style /v1/messages API endpoint (#82, #80, #74); fixes for missing or null chat-message content, Gemma 3n VLM per_layer_inputs, Qwen3.5 MTP verify attention, and batched quantized KV cache mask offsets (#91, #86, #78, #76); and a pinned 1.93.1 Rust toolchain (#87, #90).
be-free-snowboy
pushed a commit
to freestylejs/snowbrain-mlxcel
that referenced
this pull request
Oct 27, 2026
Add an Anthropic Messages-compatible API surface to mlxcel-server so clients written against the Anthropic schema can use the server without a translation shim. Ported from upstream mlx-vlm `server/anthropic.py` (PR lablup#1196) plus the tool-result image-handling fix (PR lablup#1200). New endpoints: - `POST /v1/messages` (and `/messages` alias): create a message, synchronous or SSE streaming. - `POST /v1/messages/count_tokens` (and `/messages/count_tokens` alias): prompt token count. The request schema covers the Anthropic surface: top-level `system` (string or text-block array), `messages`, `max_tokens`, `stop_sequences`, `temperature`/`top_p`/`top_k`, function `tools` + `tool_choice`, and `thinking`. Content blocks support `text`, `image` (base64 and url), `tool_use`, and `tool_result`, including image content nested inside `tool_result` blocks so vision models consume tool-produced images. Streaming emits the Anthropic SSE sequence (`message_start`, `content_block_start`/`delta`/`stop`, `message_delta`, `message_stop`, `error`) with `usage` populated on both the non-streaming response and the streaming `message_delta`. The implementation reuses the existing generation, chat-template, media, tool-call, and thinking-budget plumbing rather than duplicating the generation loop: - `types/anthropic_request.rs`, `anthropic_response.rs`, `anthropic_stream.rs`: inbound/outbound Anthropic types and the typed SSE event enum. - `anthropic_translator.rs`: Anthropic request to internal `ChatCompletionRequest` (system/messages/tools/tool_choice, image-in-tool-result), plus outbound content-block assembly, stop-reason mapping, and stop-sequence truncation, mirroring the Responses API translator. - `streaming_anthropic.rs`: the SSE channel and the content-block index state machine. - `routes/anthropic.rs`: thin handlers that call `prepare_chat_request_with_cache`, `build_generate_options`, the model provider, and the shared tool-call helpers. Both the message and count_tokens routes reject oversized tool arrays via the shared `MAX_TOOLS` bound to prevent template-rendering DoS. - Route registration in `app.rs`; module wiring in `types/mod.rs`, `mod.rs`, `routes/mod.rs`. User-facing API documentation is deferred to a follow-up; this change is the functional endpoint plus its unit tests. Verified with cargo fmt, cargo clippy (metal,accelerate, all-targets, -D warnings), and 44 new unit tests covering request translation, content-block assembly, stop-reason mapping, the tools DoS guard, and the streaming state machine.
be-free-snowboy
pushed a commit
to freestylejs/snowbrain-mlxcel
that referenced
this pull request
Oct 27, 2026
Bump version to 0.0.31. Highlights: MiniCPM-V 4.6 and RT-DETRv2 model support plus an Anthropic-style /v1/messages API endpoint (lablup#82, lablup#80, lablup#74); fixes for missing or null chat-message content, Gemma 3n VLM per_layer_inputs, Qwen3.5 MTP verify attention, and batched quantized KV cache mask offsets (lablup#91, lablup#86, lablup#78, lablup#76); and a pinned 1.93.1 Rust toolchain (lablup#87, lablup#90).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #73
Adds an Anthropic Messages-compatible API surface to mlxcel-server so clients written against the Anthropic schema can use the server without a translation shim. Ported from upstream mlx-vlm
server/anthropic.py(PR #1196) plus the tool-result image-handling fix (PR #1200).New endpoints
POST /v1/messages(and/messagesalias): create a message, synchronous or SSE streaming.POST /v1/messages/count_tokens(and/messages/count_tokensalias): prompt token count.What it covers
system(string or text-block array),messages,max_tokens,stop_sequences,temperature/top_p/top_k, functiontools+tool_choice, andthinking.text,image(base64 and url),tool_use, andtool_result, including image content nested insidetool_resultblocks so vision models consume tool-produced images.message_start,content_block_start/delta/stop,message_delta,message_stop,error) withusagepopulated on both the non-streaming response and the streamingmessage_delta.count_tokensroutes reject oversized tool arrays via the sharedMAX_TOOLSbound to prevent template-rendering DoS.Files
types/anthropic_request.rs,anthropic_response.rs,anthropic_stream.rs: inbound/outbound types + typed SSE event enum.anthropic_translator.rs: request translation, outbound content-block assembly, stop-reason mapping, stop-sequence truncation.streaming_anthropic.rs: SSE channel + content-block index state machine.routes/anthropic.rs: thin handlers; route registration inapp.rs; module wiring intypes/mod.rs,mod.rs,routes/mod.rs.Notes
User-facing API documentation is deferred to a follow-up; this PR is the functional endpoint plus tests.
Verification
cargo fmt --checkclean.cargo clippy --features metal,accelerate --all-targets -- -D warningsclean.