[Atharva-Kanherkar]

Fixes #2436

Problem

The Responses API can emit output-call items that were missing from NeMoGymResponseInputItem. Because NeMoGymResponseOutputItem shares that union, valid upstream responses containing those items failed local validation and the model proxy returned HTTP 500 after the provider had succeeded.

Changes

• Add SDK-backed wrappers for file search, web search, computer use, image generation, code interpreter, local shell, and custom tool calls.
• Preserve each upstream model's required type discriminator.
• Reject untagged output-call dictionaries before permissive union coercion can misclassify them.
• Treat every non-message model output item as a transcript output boundary.
• Keep input-only transcripts on the input side and preserve tool-only model output during simulation canonicalization.
• Add regression coverage for schema validation, all output boundary types, input-only transcripts, and tool-only responses.

Web search, file search, image generation, and code interpreter are provider-executed. Computer actions, local-shell commands, and custom calls are returned for the integrator to execute.

Behavior change: the no-match case in split_responses_input_output_items

Separately from the new item types, this changes what happens when no output boundary is found at all. Calling it out explicitly since it is not implied by the title.

The old loop read the leaked loop variable after break, so when nothing matched it fell through with i == len(items) - 1 and the final item was always classified as output. The new code initializes split_at = len(items), so a transcript with no output boundary is now treated as entirely input.

Items	Before	After
[function_call_output]	([], [function_call_output])	([function_call_output], [])
[user_message, function_call_output]	([user_message], [function_call_output])	([user_message, function_call_output], [])
[user_message, function_call]	([user_message], [function_call])	unchanged

The old result was an off-by-one: a trailing user or tool item was reported as model output even though the transcript contained no model turn. Transcripts that do contain a boundary item are unaffected, because the loop still breaks at the same index.

Worth noting that a single-item transcript of one of the newly recognized types, for example [web_search_call], returns the same pair before and after, because the old fallback happened to land on the correct split for that case. The two changes mask each other there, which is why the existing tests did not surface the difference.

Effect on the two callers

In both callers output_items becomes response.output, so the only transcripts affected are those with no output boundary anywhere.

• responses_api_agents/tau2/app.py: the first split breaks at the leading assistant message, per the comment there, so only the second split over the remainder is in scope. If that remainder is entirely user and tool items, its trailing item is no longer reported as model output. An empty output_items slices safely, so output_items[:1] and output_items[1:] still behave.
• responses_api_agents/swe_agents/app.py: only the non-replay branch calls this, since the replay branch computes its own split_at. A trajectory with no model turn now yields an empty output rather than one mislabeled trailing item.

In both cases the new result is the accurate one, and neither path raises on an empty list.

Verification

• 1580 passed, 369 deselected, 5 subtests passed in the complete non-sandbox unit suite.
• 117 passed across the response schemas, converter, model proxy, and conversational simulation suites.
• All applicable pre-commit hooks passed.
• git diff --check passed.
• Live forced web-search requests returned valid web_search_call output for gpt-4.1-mini, gpt-5.4-mini, gpt-5.6-luna, and gpt-5.6-terra; the proxy failure was reproduced before the schema fix and resolved afterward.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[Atharva-Kanherkar]

hey @bxyu-nvidia, sorry for the direct ping.

same bug as #1910, just the remaining output-call types (web search, file search, computer use, image gen, code interpreter, local shell, custom). they were missing from the union, so the proxy 500s after the provider already succeeded. filed as #2436.

tagging you since you wrote most of openai_utils.py and reviewed #1910. small diff, mostly tests. any chance you could look when you get time? no problem if not, thanks for your work here.

@cmunley1 it also needs vetting before ci can run.

[Atharva-Kanherkar]

thanks!<3