feat(server): accept Responses-native input_text / input_image parts and image-bearing function_call_output #1310

Summary

/v1/responses reuses the chat-completions content-part types, so a request written against the Responses API's own part vocabulary (input_text, input_image with a string image_url, input_file) fails to deserialize, and function_call_output.output only accepts a string, so a tool that returns a screenshot (output: [{"type":"input_image","image_url":"data:..."}]) is rejected before any model work. The change adds the native part types to the Responses request model, maps them onto the chat-completions parts the renderer already understands, and routes image parts inside a tool output into the prompt as a follow-up user image turn so the model actually sees them.

Current behavior

• src/server/types/responses_request.rs:265: ResponseInputContent::Parts(Vec<ContentPart>) where ContentPart (src/server/types/request.rs:147) is the chat-completions enum with variants text, image_url, video_url, input_audio. A part tagged input_text or input_image is a serde error, surfaced as a 400 before translation.

• src/server/types/responses_request.rs:235:

  FunctionCallOutput { call_id: String, output: String },

  An array-valued output is a deserialization error.

• src/server/responses_translator.rs:358 builds the tool turn as MessageContent::Text(output.clone()); there is no path for media inside a tool result.

• docs/responses-api.md:107 states the limitation: "OpenAI Responses-specific input_image / input_file compatibility is not complete."

• The render and media pipeline downstream is already shape-agnostic: routes/responses.rs:253 passes prepared.media from prepare_chat_request_with_cache, which collects every ContentPart::ImageUrl across the message list in order (ChatCompletionRequest::image_urls).

Expected behavior

Accepted input part shapes on message items and inside function_call_output.output arrays:

function_call_output.output accepts a string (unchanged) or an array of the parts above. Translation of an array output:

1. Text parts are joined with "\n" into the tool message content.
2. Parts that are neither text nor image are JSON-serialised (serde_json::to_string of the array of leftovers) and appended as one more text line, so nothing is silently dropped.
3. If the array held N image parts, the line [Image output attached in the next message] is appended to the tool message, and a new message {"role":"user","content":[N x ContentPart::ImageUrl]} is pushed immediately after the tool message. The image order inside the follow-up turn is the order inside output.

Images on a message item with role user stay on that turn as content parts (no reordering). Images on an assistant, system or developer message item are moved to a following user turn the same way as tool outputs, because chat templates render image placeholders only inside user turns.

Stored responses (previous_response_id / conversation) replay the translated items, so a tool output that carried images is reproduced identically when the conversation continues.

Implementation plan

1. src/server/types/responses_request.rs:
   • Add #[derive(Deserialize)] #[serde(tag = "type", rename_all = "snake_case")] pub enum ResponseInputPart { InputText { text: String }, InputImage { #[serde(default)] image_url: Option<String>, #[serde(default)] detail: Option<String>, #[serde(default)] file_id: Option<String> }, InputFile { #[serde(flatten)] raw: serde_json::Map<String, serde_json::Value> }, Text { text: String }, ImageUrl { image_url: ImageUrl }, VideoUrl { video_url: VideoUrl }, InputAudio { input_audio: InputAudio } }. The last four mirror the chat-completions ContentPart variants one to one so existing clients keep working; add impl TryFrom<ResponseInputPart> for ContentPart that performs the mapping in the table (and returns the 400 messages for file_id-only and input_file).
   • Change ResponseInputContent::Parts(Vec<ContentPart>) to Parts(Vec<ResponseInputPart>) and update as_text to read both InputText and Text.
   • Change FunctionCallOutput { call_id: String, output: ResponseToolOutput } with #[serde(untagged)] pub enum ResponseToolOutput { Text(String), Parts(Vec<ResponseInputPart>) }.
2. src/server/responses_translator.rs:
   • Add fn lower_parts(parts: &[ResponseInputPart]) -> Result<(Vec<ContentPart>, Vec<ImageUrl>), String> that maps InputText / InputImage per the table, returns the 400 message for file_id-only images and input_file, and collects the images in order.
   • In input_items_to_messages (src/server/responses_translator.rs:305 onward), for Message items: if role is User, keep the lowered parts as MessageContent::Parts; otherwise emit the text as MessageContent::Text and push the image follow-up user turn image_followup_message(images) right after.
   • For FunctionCallOutput, replace the single MessageContent::Text(output.clone()) with the three-step lowering above; push the follow-up user turn after the tool message. Keep the existing pending_reasoning clearing in that arm.
   • input_items_to_messages becomes fallible (Result<Vec<Message>, String>); the two callers in translate_request return the message as a 400 invalid_request_error through the existing error path in routes/responses.rs.
3. src/server/responses_store.rs / conversation_store.rs: the canonical stored input items are the raw ResponseInputItems (canonical_input_items at src/server/responses_translator.rs:175), so storage needs no schema change; confirm replay round-trips through the new enum with a unit test.
4. src/server/routes/responses.rs:101 (empty-input rejection): count InputImage parts and image parts inside tool outputs as effective input so an image-only request is not rejected as empty.
5. docs/responses-api.md: replace the "not complete" sentence with the table above and the image-follow-up rule for tool outputs.

Validation

(a) Unit tests in src/server/responses_translator.rs tests module and src/server/types/responses_request.rs tests module:

• input_text_and_input_image_parts_lower_to_chat_parts (user message with input_text + input_image data URI yields MessageContent::Parts with Text then ImageUrl, detail carried).
• input_image_file_id_only_is_rejected_with_named_error.
• input_file_part_is_rejected.
• function_output_image_stays_after_tool_result (tool output [input_text "done", input_image] yields a tool message whose text ends with [Image output attached in the next message] followed by a user message with exactly one ImageUrl part; the rendered prompt has the tool text before the image placeholder).
• function_output_preserves_text_alongside_visual_input.
• function_output_string_form_is_unchanged.
• unknown_function_output_parts_remain_as_json_text (a part of type foo is serialised into the tool text, images list empty).
• assistant_message_image_moves_to_following_user_turn.
• stored_response_with_image_tool_output_replays_identically (round-trip through responses_store).

(b) Real checkpoint: mlx-community/Qwen3-VL-4B-Instruct-4bit (image-capable, tool-capable).

./target/release/mlxcel-server -m models/qwen3-vl-4b-instruct-4bit --port 8080
IMG=$(base64 -i tests/fixtures/red_square.png)   # any small image
curl -s localhost:8080/v1/responses -H 'content-type: application/json' -d "{
  \"model\":\"q\",
  \"input\":[
    {\"type\":\"message\",\"role\":\"user\",\"content\":[{\"type\":\"input_text\",\"text\":\"Take a screenshot and tell me its dominant color.\"}]},
    {\"type\":\"function_call\",\"call_id\":\"c1\",\"name\":\"screenshot\",\"arguments\":\"{}\"},
    {\"type\":\"function_call_output\",\"call_id\":\"c1\",\"output\":[{\"type\":\"input_text\",\"text\":\"captured\"},{\"type\":\"input_image\",\"image_url\":\"data:image/png;base64,$IMG\"}]}
  ],
  \"tools\":[{\"type\":\"function\",\"name\":\"screenshot\",\"parameters\":{\"type\":\"object\",\"properties\":{}}}]
}" | jq -r '.output[] | select(.type=="message") | .content[0].text'

Acceptance: HTTP 200, the answer names the image's dominant colour (red for a red square), and the server log shows one image in the prepared media for the request. A second request with {"type":"input_image","file_id":"f1"} returns HTTP 400 with the named message. A user message carrying input_image directly produces the same answer as the equivalent image_url chat-completions request.

Acceptance criteria

• input_text and input_image parts deserialize on message items and inside function_call_output.output arrays.
• file_id-only images and input_file parts return a named 400.
• Image parts in tool outputs and non-user messages reach the model as a following user image turn; user-turn images keep their position.
• Stored-response replay preserves the lowered messages.
• docs/responses-api.md updated.
• Unit tests above added; real-checkpoint check above passes.
• cargo test --workspace --profile test-fast --features metal,accelerate passes
• cargo clippy --workspace --all-targets -- -D warnings and cargo fmt --all -- --check pass

Out of scope

• input_file support (PDF / document ingestion).
• file_id resolution against an uploads store.
• Merging multiple system / developer items into one leading system turn.