feat(model): bridge Responses custom tools over function-only chat backends by alexandery-nvidia · Pull Request #4880 · NVIDIA-NeMo/Gym

alexandery-nvidia · 2026-10-21T21:08:14Z

What this does

Adds custom_tool_strategy to the shared model-server config (nemo_gym/base_responses_api_model.py), with two values:

passthrough (default) keeps the current behavior, with drop_custom_tools remaining the kill switch (one deliberate exception for namespaced sub-tools, listed under Compatibility).
function_bridge lets a client that speaks the Responses custom-tool (free-form tool) dialect run against a backend whose chat schema admits function tools only. At the conversion boundary (nemo_gym/responses_converter.py), each custom tool is lowered to a function tool with one required plain-string parameter — a string-typed schema round-trips the payload byte-verbatim through schema coercion; custom_tool_call/custom_tool_call_output history items are replayed in the same lowered form; a returned call whose name the request declared as custom is lifted back into a custom_tool_call item; custom references inside tool_choice (including allowed_tools) are rewritten to the function form vLLM's named tool choice accepts; and custom entries in a streaming harness's additional_tools carrier are hoisted instead of dropped (nemo_gym/responses_streaming.py). A grammar format is dropped at the lowering, warned once per tool name: no vLLM tool parser enforces a grammar through function calling. A name declared both as a function tool and as a custom tool is rejected, because the lowering would make returned calls unattributable.

The bridged payload's wire form depends on the backend's tool parser: an XML-style parser (qwen3_coder, nemotron_v3) carries the string payload as raw text in its markup, while a JSON-format parser (hermes, llama3_json, mistral) requires the model to JSON-escape the payload inside the arguments object, so an escaping mistake surfaces as plain text with no tool call. The config description and the docs row state this.

Why

A Responses custom tool already converts to the Chat Completions custom form, but two gates stop it short of a vLLM backend: replayed custom_tool_call/custom_tool_call_output items raise NotImplementedError in the converter's input dispatch, and vLLM's chat schema admits function tools only, so the engine refuses the request before any template runs. drop_custom_tools can only remove the tools, which removes the capability. The concrete client is Codex's code mode (codex ≥ 0.154.0), whose whole tool surface is one free-form exec tool shipped inside an additional_tools input item; with the bridge, that surface runs against a plain vLLM serving stack.

Also in this change

The response lift handles a chat-native custom tool call shape under either strategy, closing a latent KeyError on backends with free-form tool support.
flatten_namespace_tools preserves each sub-tool's declared type instead of forcing function, and the namespace fold and restore cover custom_tool_call items.
Token capture learns the new item kinds: a ForTraining variant and union tags for custom_tool_call, fingerprint coverage for custom_tool_call/custom_tool_call_output (FINGERPRINT_VERSION 4), and terminal-attribution keys for the call id and free-form payload.
Every server that runs the Responses-to-Chat conversion honors the setting: vllm_model and its subclasses (the compaction and GenRM servers included), inference_provider, and azure_openai_model. The compaction server's streaming dispatch applies the same tool policy (drop_custom_tools, drop_hosted_tools, the carrier hoist) through a helper shared with the base dispatch — it previously ignored all three. vllm_model rejects function_bridge together with is_responses_native: true, whose native route renders function tools only.
The silent degradation paths now log: a malformed bridged argument shape (not JSON, a non-string payload, extra keys) warns with the tool name and call id before the raw string is delivered.
Docs: a custom_tool_strategy row in the vLLM model-server config table, cross-referenced from drop_custom_tools.

Compatibility

FINGERPRINT_VERSION moves 3 to 4 (the fingerprint space gains the custom item kinds). Hashes for existing item types are unchanged, but a v4 reader ignores stored v3 lineage/terminal entries, so a run resumed across the upgrade recomputes those rows instead of reusing them.
One deliberate passthrough-path change: a namespaced custom sub-tool in top-level tools keeps its declared type instead of being coerced to a parameterless function tool and silently dropped at validation. It then follows the same rule as a top-level custom tool: kept for a backend with free-form tool support, refused by a function-only backend unless drop_custom_tools is set. A regression test pins the shape.
Known follow-up, not in this PR: Gym's tool-call metrics and exports (rollout_collection.py tool-call counts, observability/model_calls.py, atif_export.py) recognize function_call only, so bridged rollouts report no tool calls there. Native Responses backends already hit this; the bridge makes it more common.

Testing

tests/unit_tests/test_custom_tool_function_bridge.py (new): fixture shapes mirror codex 0.154.0 code mode as captured against a mock /v1/responses endpoint — the exec custom tool with a lark grammar carried in an additional_tools developer item, bare-name custom_tool_call replays, and string or structured custom_tool_call_output. It also covers tool_choice rewrites (direct and allowed_tools), a tool without description, the both-kinds name rejection, chat-shaped custom tool_calls turns, the training-variant lift with token IDs, replay pinned against a literal engine-format arguments string (byte-equal for a well-formed call, object-equal across encoding differences, re-wrapped for a malformed one), the malformed-shape warnings, and the passthrough regression for a namespaced custom sub-tool.
Server suites: converter wiring (vllm_model and GenRM forward the strategy), the is_responses_native rejection, and the shared streaming-sanitize policy helper every /v1/responses dispatch uses.
Full tests/unit_tests/ run on this branch matches the main baseline in the same environment (no new failures; the telemetry modules need the optional opentelemetry extra in both).
End-to-end locally: codex 0.154.0 in code mode → vllm_model with custom_tool_strategy=function_bridge → a function-only engine; the lowered request and lifted response round-tripped the free-form payload byte-identically.
End-to-end on a cluster: a full codex code-mode session against a served checkpoint sustained 56 exec call/output round trips, every pair matched by call id, with both codex output encodings (success string and failure part list) crossing the bridge. NVIDIA-internal reference for that run: Slurm job 1167906, experiment codemode-smoke3 (the reconstructed rollout with the 56 round trips is artifacts/rollouts.jsonl in the experiment directory); the launcher flag and validation write-up are in the internal merge request atlas/nemorl!73.
ruff check and ruff format --check pass on the touched files.

🤖 Generated with Claude Code

…ckends A Responses custom (free-form) tool already converts to the Chat Completions custom form, but two gates stop it short of a vLLM backend: replayed custom_tool_call/custom_tool_call_output items raise NotImplementedError in the converter's input dispatch, and vLLM's chat schema admits function tools only, so the engine refuses the request before any template runs. Codex's code mode additionally ships its tools inside an additional_tools input item, from which the streaming sanitizer hoisted function entries only. This adds custom_tool_strategy to the shared model-server config: - passthrough (default) keeps every current behavior bit for bit, with drop_custom_tools remaining the kill switch. - function_bridge lowers each custom tool at the conversion boundary to a function tool with one required plain-string parameter (a string-typed schema round-trips the payload byte-verbatim through schema coercion), replays custom-tool history items in the same lowered form, lifts a returned call whose name the request declared as custom back into a custom_tool_call item, and hoists custom entries from the additional_tools carrier. A grammar format is dropped at the lowering: no vLLM tool parser enforces a grammar through function calling. The response lift also handles a chat-native custom tool call shape under either strategy, closing a latent KeyError on backends with free-form tool support. flatten_namespace_tools now preserves each sub-tool's declared type instead of forcing function, the namespace fold and restore cover custom_tool_call items, and token capture learns the new item: a ForTraining variant and union tags for custom_tool_call, fingerprint coverage for custom_tool_call/custom_tool_call_output (FINGERPRINT_VERSION 4), and terminal-attribution keys for the call_id and free-form payload. Fixture shapes in the new tests mirror codex 0.154.0 code mode as captured against a mock /v1/responses endpoint: the exec custom tool with a lark grammar carried in an additional_tools developer item, bare-name custom_tool_call replays, and string or structured custom_tool_call_output. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: alexandery <alexandery@nvidia.com>

copy-pr-bot · 2026-10-21T21:08:26Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

alexandery-nvidia · 2026-10-21T21:08:28Z

/ok to test a68d39c

yaoyu-33

Thanks for this. Bridging Codex code mode's free-form exec tool onto a function-only vLLM is a real gap, and the cluster run with 56 round trips is solid evidence. The opt-in custom_tool_strategy keeps the change contained, and the lift/replay/training-variant plumbing is consistent end to end.

Three things I'd like fixed before merge (inline):

A custom tool without a description fails to convert under function_bridge.
tool_choice / allowed_tools references to a custom tool aren't lowered along with the tool.
flatten_namespace_tools changes the default passthrough path for a namespaced custom sub-tool in top-level tools.

The rest are notes: config wiring on servers other than vllm_model, docs, a circular replay test, stale comments, and a few test gaps.

PR body / compatibility

FINGERPRINT_VERSION 3 → 4 is global. Hashes for existing item types are unchanged, but stored v3 lineage/terminal entries are now ignored, so a run resumed across the upgrade falls back for those rows. Could you add a line about this under compatibility?
The body says passthrough keeps behavior "bit for bit"; see the inline note on responses_streaming.py:76.
Not blocking, possible follow-up: Gym's own tool-call metrics and exports (rollout_collection.py tool-call counts, observability/model_calls.py, atif_export.py) only recognize function_call. Bridged rollouts will report no tool calls there, and ATIF export may reject them. Native Responses backends already hit this; the bridge just makes it more common.

Validation done for this review: the new test file plus the related converter, streaming, fingerprint, terminal and openai_utils suites pass locally (CPU). The failure cases in the inline comments were each reproduced by calling the converter/sanitizer directly. Not verified: GPU or a real vLLM backend.

ananthsub · 2026-10-22T18:03:36Z

This builds on @yaoyu-33's review and only adds what it doesn't already cover.

One more should-fix: vllm_model_with_compaction drops Codex's exec tool from the additional_tools item even with function_bridge set, because its own responses_dispatch never passes the flag to the streaming sanitizer (reply on the base_responses_api_model.py:517 thread).

The rest are smaller:

The config description says natively served Responses carry custom tools, which isn't true on vLLM with is_responses_native: true.
The "no JSON-escaping burden" comment holds only for XML-style tool parsers.
The unbridge fallback and the grammar drop are both silent.
A function tool and a custom tool with the same name collide.
A few naming and comment fixes.

Two points where this review disagrees with the existing comments, with the reasons in the threads:

On responses_streaming.py:76, keeping the declared sub-tool type under passthrough matches how a top-level custom tool already behaves, so correcting the PR body seems better than gating it on carry_custom_tools.
On the replay test, vLLM parses assistant tool-call arguments into a dict before rendering the template, so separator and escaping differences in the re-encoded string don't change the prompt; the double wrapping of a malformed call still does.

linj-glitch · 2026-10-22T20:30:46Z

LGTM once all comments resolved.

Review fixes for #4880: - Lower custom references in tool_choice (and allowed_tools) to the function form the backend's named tool choice accepts. - Carry a custom tool's optional description only when present. - Reject a name declared both as a function and as a custom tool under the bridge: the lowering would make returned calls unattributable. - Forward custom_tool_strategy in the GenRM, inference_provider and azure_openai_model converters, and route every streaming /v1/responses dispatch (the compaction server's included) through one base-class helper that applies the config's tool policy. - Reject function_bridge together with is_responses_native on vllm_model: the native route skips the conversion that implements the bridge. - Warn on the grammar-format drop (once per tool name) and on each malformed bridged-argument shape, with the tool name and call id. - Name the bridge helpers private with one verb pattern, share one CustomToolStrategy alias between the converter and the config, and rename the sanitizer flag to hoist_carrier_custom_tools. - Docs: a custom_tool_strategy row in the vLLM config table with the tool-parser escaping caveat; stale comments brought up to date. - Tests: tool_choice rewrites, a tool without description, the name collision, replay pinned against a literal engine-format arguments string, the training-variant lift, chat-shaped custom turns, the passthrough namespaced sub-tool regression, converter wiring and the shared sanitize-policy helper. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: alexandery <alexandery@nvidia.com>

alexandery-nvidia · 2026-10-22T21:24:03Z

/ok to test 10854c3

alexandery-nvidia · 2026-10-22T21:25:55Z

All review comments are addressed in 10854c3, with a reply in each thread. The PR body now carries a Compatibility section: the FINGERPRINT_VERSION 3-to-4 note, the corrected passthrough claim, and the tool-call-metrics follow-up.

One thread is left open on purpose — the responses_streaming.py:76 one, where the two reviews disagree about gating the namespaced sub-tool's kept type on function_bridge versus keeping it unconditional. The code currently keeps it unconditional, a regression test pins that shape, and the body documents it under Compatibility; flipping to the gate is a small change if that's the call.

CI: /ok to test is posted on 10854c3. The "Harness conformance (non-blocking)" probe failures (codex, opencode, pi, kilocode) reproduce on every current PR-context run of that workflow and pass in merge-queue runs, so they predate this branch — hermes and stirrup pass here through the same converted path.

github-actions · 2026-10-22T21:32:35Z

🌿 Preview your docs: https://nvidia-preview-pr-4880-20c3db232d31.docs.buildwithfern.com/nemo/gym

yaoyu-33 · 2026-10-23T19:11:09Z

/ok to test 2ca9955

alexandery-nvidia · 2026-10-23T19:58:40Z

/ok to test 20c3db2

github-actions · 2026-10-23T22:12:25Z

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40032076896

NVIDIA-NeMo#5155) * feat(responses): opt-in keepalive events for long streaming responses Streaming /v1/responses requests are buffered: the converted response is re-emitted as SSE only once responses() returns, so the client sees no event while a long generation runs. A client that bounds the silence between events abandons the turn and re-sends it. The Codex CLI does this after stream_idle_timeout_ms (300 s by default), which a high reasoning effort turn can easily outlast. SSE comments, which the chat streaming path sends, do not help here: Codex's event parser drops them before its idle timer sees them. Add stream_keepalive_s to BaseResponsesAPIModelConfig (default null, today's behavior). When set and a call is still running after that many seconds, the stream opens with response.created and then carries response.in_progress every interval for the same placeholder response id. Once responses() returns, the item and terminal events follow exactly as the buffered path sends them, minus the response.created the keepalives already sent. A call that finishes within the first interval streams exactly as before, HTTP error statuses included. A failure after the stream opened ends with response.failed for the announced response id, carrying what the client would have seen before the stream opened: a refusal the server propagates keeps its OpenAI code (context_length_exceeded), an upstream HTTP error follows the server's propagation policy, and no upstream request URL is reported. Model-call capture measured time to first token from the first body chunk, which with keepalives would be the opening response.created one interval in. The Responses lifecycle events (response.created and response.in_progress) carry no model output, so they no longer count toward it; on the buffered path they are sent in the same instant as the first item, so its measurement is unchanged. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> * feat(vllm_model): accept unlisted reasoning efforts and all_turns context The Codex CLI sends reasoning efforts its model catalog offers, and a catalog may offer one the pinned OpenAI SDK does not list, such as "max". Request validation types reasoning.effort with the SDK's literal, so vllm_model answers such a request with 422 before any conversion, although the engine behind the chat route accepts the level. Add forward_unvalidated_reasoning_effort to VLLMModelConfig (default false, today's behavior; requires is_responses_native=false). When set, a string effort the SDK does not list is taken out of the raw Responses body before validation and put verbatim on the converted chat request as reasoning_effort; the engine validates it. Efforts the SDK lists, "none" included, keep their existing path, and an empty effort is still refused. Codex also sends reasoning.context "all_turns" when its catalog enables reasoning context, which drop_unrepresentable_request_fields still refuses. The conversion passes every reasoning item the request carries to the engine and leaves what is rendered to its chat template, which is what "all_turns" (render them all) and "auto" (the provider's choice) ask for. With drop_unrepresentable_request_fields those two values are now dropped like the summary fields. "current_turn" asks for earlier turns' reasoning to be left out, which the conversion does not do, so it is still refused, and every value is still refused by default. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> * feat(vllm_model): bridge custom tools and inline tool output images Two things the Codex CLI sends cannot reach a vLLM chat route through the Responses to Chat Completions conversion today. Codex's apply_patch is a Responses custom (freeform) tool with a lark grammar. The conversion forwards it as a Chat Completions custom tool, which a vLLM chat route rejects, so the only option was drop_custom_tools: the model then has no apply_patch, tries it in the shell, and falls back to editing with sed. Add custom_tools_as_functions (VLLMModelConfig and ResponsesConverter, default false). Each custom tool is presented to the engine as a function tool with one required string parameter, input; its description is the tool's description, a note to pass the raw input as input, and the grammar when the tool has one, since a string parameter cannot constrain the input. The engine's calls to that function come back as custom_tool_call items (call_id, name, input, status) on the non-streaming and the streaming path alike. Arguments that are not an object with a string input are passed through as the input, so the client's tool reports the bad call to the model. Replayed custom_tool_call and custom_tool_call_output items become the chat tool call (arguments {"input": ...}) and tool message, on the same assistant turn as the replayed reasoning. A named tool_choice of a custom tool selects its function. The option takes precedence over drop_custom_tools, and a custom tool named like a function tool fails the request. It needs is_responses_native=false and return_token_id_information=false: custom_tool_call has no variant that carries token IDs. Saved model-call records now list custom_tool_call items among a response's tool calls. When the model catalog lists image input, Codex offers view_image, whose output is a function_call_output holding an input_image part. Chat tool messages accept image parts since NVIDIA-NeMo#4688, but the conversion still refuses the part, or replaces it with a text placeholder under drop_unrepresentable_request_fields, so the model never sees the image. Add inline_tool_output_images (VLLMModelConfig and ResponsesConverter, default false; needs is_responses_native=false). Each image part of a tool call output becomes an image_url part of the chat tool message, which vLLM accepts and leaves to the chat template, and it takes precedence over the placeholder. Detail "original", which Codex sends when the model asks view_image for full resolution and which chat lacks, goes as "high". Other part types a tool message cannot carry are still refused. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> * fix(vllm_model): reject forward_unvalidated_reasoning_effort where it cannot apply The completions path builds its body from a fixed list of chat fields, and reasoning_effort is not one of them, so with use_completions_api=true the forwarded effort was dropped without a word. The compaction server has its own dispatch, which never sets the unlisted effort aside. Both now refuse the option at startup instead of ignoring it. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> * refactor(vllm_model): use main's function bridge for custom tools NVIDIA-NeMo#4880 added custom_tool_strategy: function_bridge, which lowers each Responses custom tool to a function tool with one string input and lifts the calls back, on the same conversion this branch extended. Drop this branch's custom_tools_as_functions, which did the same job: the config field, the converter helpers and restore step, the name-clash check, the streaming drop override and their tests. The Codex tests now run apply_patch through function_bridge, streaming and non-streaming, with drop_custom_tools off: it is still the kill switch and would drop the tool before the bridge sees it. Saved model-call records keep listing custom_tool_call items, which the bridge returns too. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> * fix(vllm_model): measure context with the reasoning effort generation forwards Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> * fix(checkpoint): treat Responses lifecycle events as liveness Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> * test(vllm_model): let Responses lifecycle events precede publication Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> * fix(model): keep one response id across a keepalive stream Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> * fix(genrm_model): forward inline_tool_output_images to the converter Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> --------- Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> Co-authored-by: gwarmstrong <gwarmstrong@users.noreply.github.com>

…eMo#5219) * feat(responses): opt-in keepalive events for long streaming responses Streaming /v1/responses requests are buffered: the converted response is re-emitted as SSE only once responses() returns, so the client sees no event while a long generation runs. A client that bounds the silence between events abandons the turn and re-sends it. The Codex CLI does this after stream_idle_timeout_ms (300 s by default), which a high reasoning effort turn can easily outlast. SSE comments, which the chat streaming path sends, do not help here: Codex's event parser drops them before its idle timer sees them. Add stream_keepalive_s to BaseResponsesAPIModelConfig (default null, today's behavior). When set and a call is still running after that many seconds, the stream opens with response.created and then carries response.in_progress every interval for the same placeholder response id. Once responses() returns, the item and terminal events follow exactly as the buffered path sends them, minus the response.created the keepalives already sent. A call that finishes within the first interval streams exactly as before, HTTP error statuses included. A failure after the stream opened ends with response.failed for the announced response id, carrying what the client would have seen before the stream opened: a refusal the server propagates keeps its OpenAI code (context_length_exceeded), an upstream HTTP error follows the server's propagation policy, and no upstream request URL is reported. Model-call capture measured time to first token from the first body chunk, which with keepalives would be the opening response.created one interval in. The Responses lifecycle events (response.created and response.in_progress) carry no model output, so they no longer count toward it; on the buffered path they are sent in the same instant as the first item, so its measurement is unchanged. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> * feat(vllm_model): accept unlisted reasoning efforts and all_turns context The Codex CLI sends reasoning efforts its model catalog offers, and a catalog may offer one the pinned OpenAI SDK does not list, such as "max". Request validation types reasoning.effort with the SDK's literal, so vllm_model answers such a request with 422 before any conversion, although the engine behind the chat route accepts the level. Add forward_unvalidated_reasoning_effort to VLLMModelConfig (default false, today's behavior; requires is_responses_native=false). When set, a string effort the SDK does not list is taken out of the raw Responses body before validation and put verbatim on the converted chat request as reasoning_effort; the engine validates it. Efforts the SDK lists, "none" included, keep their existing path, and an empty effort is still refused. Codex also sends reasoning.context "all_turns" when its catalog enables reasoning context, which drop_unrepresentable_request_fields still refuses. The conversion passes every reasoning item the request carries to the engine and leaves what is rendered to its chat template, which is what "all_turns" (render them all) and "auto" (the provider's choice) ask for. With drop_unrepresentable_request_fields those two values are now dropped like the summary fields. "current_turn" asks for earlier turns' reasoning to be left out, which the conversion does not do, so it is still refused, and every value is still refused by default. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> * feat(vllm_model): bridge custom tools and inline tool output images Two things the Codex CLI sends cannot reach a vLLM chat route through the Responses to Chat Completions conversion today. Codex's apply_patch is a Responses custom (freeform) tool with a lark grammar. The conversion forwards it as a Chat Completions custom tool, which a vLLM chat route rejects, so the only option was drop_custom_tools: the model then has no apply_patch, tries it in the shell, and falls back to editing with sed. Add custom_tools_as_functions (VLLMModelConfig and ResponsesConverter, default false). Each custom tool is presented to the engine as a function tool with one required string parameter, input; its description is the tool's description, a note to pass the raw input as input, and the grammar when the tool has one, since a string parameter cannot constrain the input. The engine's calls to that function come back as custom_tool_call items (call_id, name, input, status) on the non-streaming and the streaming path alike. Arguments that are not an object with a string input are passed through as the input, so the client's tool reports the bad call to the model. Replayed custom_tool_call and custom_tool_call_output items become the chat tool call (arguments {"input": ...}) and tool message, on the same assistant turn as the replayed reasoning. A named tool_choice of a custom tool selects its function. The option takes precedence over drop_custom_tools, and a custom tool named like a function tool fails the request. It needs is_responses_native=false and return_token_id_information=false: custom_tool_call has no variant that carries token IDs. Saved model-call records now list custom_tool_call items among a response's tool calls. When the model catalog lists image input, Codex offers view_image, whose output is a function_call_output holding an input_image part. Chat tool messages accept image parts since NVIDIA-NeMo#4688, but the conversion still refuses the part, or replaces it with a text placeholder under drop_unrepresentable_request_fields, so the model never sees the image. Add inline_tool_output_images (VLLMModelConfig and ResponsesConverter, default false; needs is_responses_native=false). Each image part of a tool call output becomes an image_url part of the chat tool message, which vLLM accepts and leaves to the chat template, and it takes precedence over the placeholder. Detail "original", which Codex sends when the model asks view_image for full resolution and which chat lacks, goes as "high". Other part types a tool message cannot carry are still refused. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> * fix(vllm_model): reject forward_unvalidated_reasoning_effort where it cannot apply The completions path builds its body from a fixed list of chat fields, and reasoning_effort is not one of them, so with use_completions_api=true the forwarded effort was dropped without a word. The compaction server has its own dispatch, which never sets the unlisted effort aside. Both now refuse the option at startup instead of ignoring it. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> * feat(codex): opt-ins to launch Codex in a sandbox as a plain codex exec does Evaluating Codex in a task sandbox should send the requests the Codex CLI sends when a harness such as Harbor 0.24 launches `codex exec` directly in the task container. Sandbox sessions refuse extra_config, so add explicit settings. Every default keeps the current behavior: - model_server_forwards_reasoning_effort: sandbox sessions keep rejecting reasoning_effort by default, because most Gym model servers cannot guarantee it reaches the backend. The flag records the operator's check that the model server forwards Codex's `reasoning` field unchanged; the effective value is then visible in captured model calls. - inherit_sandbox_environment: give Codex the sandbox's whole environment instead of PATH, the locale, HOME and the XDG directories, keeping only CODEX_HOME private. Task images set variables such as VIRTUAL_ENV, PYTHONHASHSEED or OMP_NUM_THREADS that a harness started in the container sees. Intended for remote providers, where that environment is the task's. - features: the generated [features] table, which turns off multi_agent and code_mode by default. A null entry is left out, so the CLI default applies, as in a launch without a Gym config. - ephemeral: false drops --ephemeral, so Codex keeps session state and offers its goal tools (create_goal, get_goal, update_goal). A correlated local rollout no longer tries to remove a flag that is not there. - request_max_retries, stream_max_retries: the provider's resends before a stream starts and reconnects after one drops, beside stream_idle_timeout_ms; null leaves Codex's own defaults. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> * feat(codex): grade a failed run while its sandbox still runs Harbor runs the verifier after any agent exit that is not a timeout: a trial whose agent died while its sandbox still answers is graded, not rerun. codex_agent instead raised an execution error for every failed Codex run, so the Environment Server skipped verification and the episode was retried from scratch: an agent the kernel's OOM kill hit (once the sandbox kills a single process instead of itself), an agent whose own pkill hit the runner, or a Codex exit 1. verify_after_agent_error (off by default) returns such a run as a `failed` response, with the error in metadata.graded_agent_error, which Resources grades as a nonzero agent exit. It applies only when the supervisor confirmed cleanup, the failure that ended the run names no model-endpoint transport or 429/5xx error, and the sandbox still reports running and answers a probe command. A lost or unresponsive sandbox, an unconfirmed cleanup and an endpoint outage stay execution errors and are retried. The endpoint check reads only the terminal failure after the run's last model output (turn.failed and the last error event before it, or the last event when Codex was killed without turn.failed), so a retry that recovered no longer turns a later unrelated failure into a rerun. It recognizes Codex 0.154.0's messages: unexpected status 429/5xx, the "high demand" message for a 500, exceeded retry limit, the waiting-for-network notice, and stream disconnects from a transport error, a closed stream, a failed send or the SSE idle timeout. The output-cap disconnect (reason: max_output_tokens) stays gradable. raise_oom_score_adj (off by default) makes Codex the OOM killer's preferred victim in its sandbox. In Docker, the leaderboard's runtime, the daemon that runs the agent lives outside the container's memory cgroup, so an OOM kill can only pick the agent's own processes. In an OpenSandbox sandbox the exec daemon and Gym's supervisor share the cgroup and the oom_score_adj with Codex. The runner raises its own score, which Codex and every tool inherit, to 1000, or to 0 when the sandbox starts it below zero (a Guaranteed pod's -997), so the sandbox never becomes the node's first victim in a global OOM. The exec daemon and the supervisor keep the sandbox's own score. TB4's resources-session test covers the `failed` response status: it is graded as a nonzero agent exit. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> * feat(codex): keep session rollouts in the trial dir codex_agent removes its session directory, and with it CODEX_HOME, when a sandbox session closes. Without the model-call capture, Gym then keeps no per-call token counts for a Codex run: the response usage comes from turn.completed, which a run stopped at its time limit never emits. With the capture on, every request payload is also copied into the rollout row (one image-heavy trial's row was 2.5 GB after 20 minutes), which a full 210-trial run cannot hold in memory or on disk. New sandbox_logs_dir setting: after Codex stops, the session rollouts (CODEX_HOME/sessions/**/rollout-*.jsonl, with per-call token counts and timestamps) are copied to <sandbox_logs_dir>/sessions before the session directory is removed. A failed copy is logged and never fails collection. Harbor's Codex agent copies them to /logs/agent/sessions the same way; with /logs/agent, the TB4 resources server copies them into the trial directory (agent/sessions/). It only acts without --ephemeral. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> * fix(codex): run tasks whose workdir is the root A task image without WORKDIR (some Terminal-Bench-Science tasks) lends "/" as its workdir. codex_agent refused it with HTTP 422 ("runtime/session files must be outside SandboxAccess.workdir"), so every trial of those tasks failed at session start, while Harbor runs the agent there. The root holds every path, so it is accepted. The runtime and session files stay under /tmp, and any other workdir must still be disjoint from them: /tmp itself and paths under the adapter's /tmp prefixes are refused as before, and the in-sandbox check still rejects a workdir that contains or lies inside an adapter path after resolving symlinks. Tests: six new cases in codex_agent/tests/test_sandbox_sessions.py cover the root workdir (in-sandbox check and seeding) and the workdirs that are still refused. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> * refactor(vllm_model): use main's function bridge for custom tools NVIDIA-NeMo#4880 added custom_tool_strategy: function_bridge, which lowers each Responses custom tool to a function tool with one string input and lifts the calls back, on the same conversion this branch extended. Drop this branch's custom_tools_as_functions, which did the same job: the config field, the converter helpers and restore step, the name-clash check, the streaming drop override and their tests. The Codex tests now run apply_patch through function_bridge, streaming and non-streaming, with drop_custom_tools off: it is still the kill switch and would drop the tool before the bridge sees it. Saved model-call records keep listing custom_tool_call items, which the bridge returns too. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> * fix(codex): drop the ephemeral setting now that sandbox sessions keep session state Main's sandbox sessions always run Codex without --ephemeral, to keep its native session transcript, so the setting no longer changes them. Local runs keep main's --ephemeral. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> * feat(tb4): lend task sandboxes through SandboxAccess sessions Serve the shared resources-session protocol so harnesses that borrow a task sandbox (codex_agent behind single_agent_turn or single_agent_turn_legacy) can run TB4-pinned datasets, as NVIDIA-NeMo#3902 does for Terminal-Bench 2.1: - /seed_session with a ResourcesSeedSessionRequest provisions the task through TB4's existing seed lifecycle and returns SandboxAccess: a direct connection through the new `sandbox_provider_ref` block and the task workdir as its user sees it. Seeds are idempotent per resources_session_id. - /verify with the task row and the agent response grades once through the existing finalizer (artifact collection, separate offline verifier, test counts and partial credit, EFS logs) and replays the verdict on identical retries. Termination follows response.status. - /close_session fences the session before it waits, as Terminal-Bench 2.1 does: a seed still provisioning releases its sandboxes instead of lending them, even when the close's caller gives up. It then waits for provisioning or grading, releases anything still running without grading, fences late seeds, and fails (keeping the fence) when a sandbox cannot be stopped. - Completed close records expire after session_close_retry_window_seconds (default 300, the Terminal-Bench 2.1 setting), so the fence set no longer grows with every episode. Failed or pending cleanup never expires, and close retries do not extend the window. Tasks that declare agent.user are rejected, since the borrower runs as the sandbox default user. TB4's own mini-SWE seed/verify contract is unchanged. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> (cherry picked from commit 1775136793828102e9826467f67736d9849b1ef8) Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> * docs(tb4): grade an agent that died while its sandbox runs as a nonzero exit A harness that returns a failed response while its sandbox still runs (codex_agent's verify_after_agent_error) is graded with termination nonzero_exit. Document it and cover it in the sessions test. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> (cherry picked from commit a61790adffe459b9c1153887134f1f4cd78c065c) Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> * feat(terminal-bench-science): add a Codex profile and a DeepSeek V4.1 Flash run config The public leaderboard's DeepSeek V4.1 Flash and GPT-6 Astra rows ran Codex at reasoning effort max. Add `terminal_bench_science/codex`: the resources server lends each task sandbox through SandboxAccess, codex_agent installs and runs Codex 0.155.1 there, and single_agent_turn_legacy seeds both, closes Codex before grading, then closes Resources. Grading, partial credit, artifacts and EFS logs are the same as for mini-SWE. - Codex's timeout is the task's 8-hour agent budget; prepare_codex.py checks that every selected task has it. The episode deadline covers provisioning, installation, the budget and the slowest verifier. - Codex sees the task image's environment and HOME, as in Harbor. - The model name Codex sees, its version and its reasoning effort are run overrides; effort also needs model_server_forwards_reasoning_effort. Codex's provider retries keep the CLI defaults unless tbsci_codex_request_max_retries / tbsci_codex_stream_max_retries set them. - A Codex run that dies while its sandbox still runs is graded as a nonzero agent exit, as Harbor grades it; Codex and its tools are the OOM killer's preferred victims; and without --ephemeral Codex's session rollouts are kept under /logs/agent/sessions, which the trial directory collects. - prepare_codex.py writes rows whose prompt is the package's instruction.md, read and hash-checked by the resources server's loader, because an Environment Server passes the row's prompt to the agent. prepare.py now exposes its task selection. codex_deepseek_flash.yaml is the run config for the leaderboard's DeepSeek V4.1 Flash / Codex / effort max row with a self-hosted model: Codex 0.154.0 with no model catalog, effort max, the full 28800 s budget, and an OpenAI Responses endpoint serving deepseek-flash in front of a chat-completions server; the endpoint URL and its key are overrides (--model-url; POLICY_API_KEY or ++tbsci_model_api_key_env). openai_model relays Codex's requests unchanged and adds no time limit of its own, and the agent keeps Codex's feature defaults and session state, so the first request matches a plain `codex exec` launch as Harbor 0.24 starts it: same instructions, input items, reasoning settings and tools, including the multi-agent and goal tools. The README documents both setups, including the leaderboard's catalog model and effort max through openai_model's responses_stream_passthrough. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> (cherry picked from commit d131d1d30391cefb7c2c2ca0a4b17758501ba6db) Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> * fix(terminal-bench-science): match the Codex profile to the updated agents - Run one trial per task by default, as the mini-SWE profile does; ++num_repeats=3 runs the leaderboard's three. - Drop codex_agent's ephemeral setting: sandbox sessions always run Codex without --ephemeral. - Name the vllm_model settings Codex needs, with NVIDIA-NeMo#4880's custom_tool_strategy: function_bridge. - Describe how a provider failure Codex reports itself reaches the outcome policy (NVIDIA-NeMo#4834). - Check the Codex manifest against its runtime composition. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> (cherry picked from commit 34cb6349bb10531fdc28f66a50a928fbdaf894a7) Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> * feat(terminal-bench-science): serve the DeepSeek Codex run through vllm_model Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> * fix(vllm_model): measure context with the reasoning effort generation forwards Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> * fix(codex_agent): retry Codex's capacity and request-timeout endpoint errors Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> * fix(codex_agent): keep usage and observations of a graded failed run Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> * fix(codex_agent): fit the session rollout copy inside the capture budget Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> * fix(checkpoint): treat Responses lifecycle events as liveness Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> * test(vllm_model): let Responses lifecycle events precede publication Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> * fix(model): keep one response id across a keepalive stream Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> * fix(genrm_model): forward inline_tool_output_images to the converter Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> * fix(terminal_bench_4): finish close cleanup after its caller disconnects Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> * fix(terminal_bench_4): retry a failed sandbox teardown on the next close Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> * fix(terminal_bench_4): mask the verify of a lost or expired resources session Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> * docs(terminal_bench_4): describe close teardown retries Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> * docs: link the Terminal-Bench-Science Codex profile from the harness guide Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> * docs(terminal_bench_science): replace a PR placeholder with NVIDIA-NeMo#5155 Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> * fix(terminal_bench_science): prepare Codex rows through gym eval prepare Without pathspec, which Gym's own environment leaves out, prepare_codex runs itself in the terminal_bench_4 server's environment, setting that up first as gym env start does. The selection is checked before the hand-off. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> --------- Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> Co-authored-by: gwarmstrong <gwarmstrong@users.noreply.github.com>

svcnvidia-nemo-ci added area:model Model servers, inference providers, and model adapters feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer labels Oct 21, 2026

nemo-automation-bot Bot requested review from a team October 21, 2026 21:09

copy-pr-bot Bot temporarily deployed to public October 21, 2026 21:09 Inactive

copy-pr-bot Bot temporarily deployed to public October 21, 2026 21:11 Inactive

copy-pr-bot Bot temporarily deployed to public October 21, 2026 21:12 Inactive

alexandery-nvidia requested review from terrykong and removed request for terrykong October 22, 2026 17:02

yaoyu-33 reviewed Oct 22, 2026

View reviewed changes

ananthsub reviewed Oct 22, 2026

View reviewed changes

linj-glitch previously approved these changes Oct 22, 2026

View reviewed changes

alexandery-nvidia dismissed linj-glitch’s stale review via 10854c3 October 22, 2026 21:23

nemo-automation-bot Bot requested review from a team October 22, 2026 21:30

copy-pr-bot Bot temporarily deployed to public October 22, 2026 21:30 Inactive

copy-pr-bot Bot temporarily deployed to public October 22, 2026 21:40 Inactive

copy-pr-bot Bot temporarily deployed to public October 22, 2026 21:41 Inactive

yaoyu-33 approved these changes Oct 23, 2026

View reviewed changes

Merge branch 'main' into alexandery/responses-custom-tool-bridge

2ca9955

nemo-automation-bot Bot requested a review from a team October 23, 2026 19:11

copy-pr-bot Bot temporarily deployed to public October 23, 2026 19:11 Inactive

copy-pr-bot Bot temporarily deployed to public October 23, 2026 19:13 Inactive

copy-pr-bot Bot temporarily deployed to public October 23, 2026 19:14 Inactive

Merge branch 'main' into alexandery/responses-custom-tool-bridge

20c3db2

alexandery-nvidia enabled auto-merge October 23, 2026 19:21

copy-pr-bot Bot temporarily deployed to public October 23, 2026 19:59 Inactive

copy-pr-bot Bot temporarily deployed to public October 23, 2026 20:01 Inactive

alexandery-nvidia added this pull request to the merge queue Oct 23, 2026

yaoyu-33 added ready-to-merge PR is approved, current, and waiting for final CI or merge and removed needs-review PR is ready for code review and waiting on a reviewer labels Oct 23, 2026

alexandery-nvidia removed this pull request from the merge queue due to a manual request Oct 23, 2026

alexandery-nvidia added this pull request to the merge queue Oct 23, 2026

Merged via the queue into main with commit aa53f3b Oct 23, 2026
83 of 88 checks passed

alexandery-nvidia deleted the alexandery/responses-custom-tool-bridge branch October 23, 2026 23:05

This was referenced Oct 27, 2026

feat(vllm_model): serve Codex through the Responses-to-chat conversion #5155

Merged

feat(terminal-bench-science): run Codex in the task sandbox #5219

Merged

gwarmstrong mentioned this pull request Nov 2, 2026

feat(openai_model): relay streaming Responses calls unchanged #5457

Open

Conversation

alexandery-nvidia commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this does

Why

Also in this change

Compatibility

Testing

Uh oh!

copy-pr-bot Bot commented Oct 21, 2026

Uh oh!

alexandery-nvidia commented Oct 21, 2026

Uh oh!

yaoyu-33 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ananthsub commented Oct 22, 2026

Uh oh!

linj-glitch commented Oct 22, 2026

Uh oh!

alexandery-nvidia commented Oct 22, 2026

Uh oh!

alexandery-nvidia commented Oct 22, 2026

Uh oh!

github-actions Bot commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yaoyu-33 commented Oct 23, 2026

Uh oh!

alexandery-nvidia commented Oct 23, 2026

Uh oh!

github-actions Bot commented Oct 23, 2026

Uh oh!

Uh oh!

This branch was previously deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

alexandery-nvidia commented Oct 21, 2026 •

edited

Loading

github-actions Bot commented Oct 22, 2026 •

edited

Loading