feat: add inference_provider server for hosted inference providers by cwing-nvidia · Pull Request #1286 · NVIDIA-NeMo/Gym

cwing-nvidia · 2026-05-10T22:19:09Z

Summary

Extract the Responses ↔ Chat Completions converter from vllm_model into a shared module (nemo_gym/responses_converter.py) so it can be reused without pulling in vLLM-specific logic
Add a new chat_completions_model server for eval workloads against any provider that exposes an OpenAI-compatible /v1/chat/completions endpoint
Include convenience YAML configs for: Fireworks, Together.ai, OpenRouter, DeepInfra, Nebius, Friendli, Baseten, HF Inference, and Gemini

Motivation

Many users want to evaluate benchmarks against models hosted on inference providers without self-hosting vLLM. The existing vllm_model works but carries vLLM-specific baggage (token ID support, /tokenize calls, chat_template_kwargs) that is irrelevant and potentially error-prone for hosted providers.

Note

Excluded nemo_gym/responses_converter.py from core unit test coverage
(tested by server-level tests, same as before extraction from vllm_model)

Test plan

Closes #1285
Ref #1147

Extract the Responses <-> Chat Completions converter from vllm_model into a shared module (nemo_gym/responses_converter.py) and introduce a new chat_completions_model server for eval workloads against any provider that exposes an OpenAI-compatible /v1/chat/completions endpoint. Includes convenience configs for: Fireworks, Together.ai, OpenRouter, DeepInfra, Nebius, Friendli, Baseten, HF Inference, and Gemini. Closes #1285 Ref #1147 Signed-off-by: Chris Wing <cwing@nvidia.com> Co-authored-by: Cursor <cursoragent@cursor.com>

copy-pr-bot · 2026-05-10T22:19:13Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

- Exclude nemo_gym/responses_converter.py from core unit test coverage (tested by server-level tests, same as before extraction from vllm_model) - Add pragma: allowlist secret to test dummy API key Signed-off-by: Chris Wing <cwing@nvidia.com> Co-authored-by: Cursor <cursoragent@cursor.com>

After extracting ResponsesConverter to nemo_gym/responses_converter.py, the uuid4 monkeypatch must target the new module path. Signed-off-by: Chris Wing <cwing@nvidia.com> Co-authored-by: Cursor <cursoragent@cursor.com>

cmunley1 · 2026-05-29T21:48:05Z

I think the naming is pretty confusing, because vllm and openai also support chat completions endpoint. Any thought to rename this?

cmunley1 · 2026-05-29T21:48:32Z

It also seems to clash with responses_api_model/chat_completions_model -- feels strange

Names the server by use case (hosted inference providers) rather than the API endpoint shared by all model servers. Signed-off-by: Chris Wing <cwing@nvidia.com>

Keep both coverage omit entries: responses_converter.py (PR) and environments/*/prepare.py (main). Signed-off-by: Chris Wing <cwing@nvidia.com>

Merge with main dropped the pydantic Field import while keeping default_headers on VLLMModelConfig. Signed-off-by: Chris Wing <cwing@nvidia.com>

cwing-nvidia · 2026-06-01T06:30:14Z

I think the naming is pretty confusing, because vllm and openai also support chat completions endpoint. Any thought to rename this?

@cmunley1 agreed - updated the naming to inference_provider. @kajalj22 can you re-review?

…1289) ## Summary - Add `inference-providers.mdx` — docs page for the `inference_provider` server covering setup, config reference, usage example - Rewrite `model-server/index.mdx` — rename to "Configure Model", simplify intro to focus on Responses API middleware story, add cloud/self-hosted/training badges, remove implementation details ## Notes - Usage example follows the `mcqa` environment chosen in the updated quickstart (#1283) - Depends on #1286 (which adds the `inference_provider` server code and provider configs) ## Test plan - [ ] `fern docs dev` — verify both pages render correctly - [ ] Tabs component renders on inference-providers page - [ ] Badge components render on index page - [ ] All internal links resolve --------- Signed-off-by: Chris Wing <cwing@nvidia.com> Signed-off-by: Lawrence Lane <llane@nvidia.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Lawrence Lane <llane@nvidia.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

github-actions · 2026-06-16T17:40:59Z

🌿 Preview your docs: https://nvidia-preview-cwing-chat-completions-model.docs.buildwithfern.com/nemo/gym

Here are the markdown pages you've updated:

Signed-off-by: Lin Jia <linj@nvidia.com> # Conflicts: # fern/versions/latest/pages/model-server/vllm.mdx

copy-pr-bot · 2026-06-22T19:21:37Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

- Make the model-server index cards consistent on a cloud vs self-hosted axis, with gym-managed/training as secondary qualifiers; add the missing Local vLLM Proxy card and sharpen the Inference Providers and vLLM blurbs - Drop the redundant policy_* note from the inference providers config reference (the env.yaml interpolation convention is repo-wide and documented centrally) - Fix Local vLLM / Local vLLM Proxy nav ordering Signed-off-by: Chris Wing <cwing@nvidia.com>

…se Gym… (#1627) # What does this PR do? [Design note](#1620 (comment)) Makes **every NeMo Gym model server speak the Anthropic Messages API** by adding a `POST /v1/messages` route to the base `SimpleResponsesAPIModel`, with a default implementation that maps Anthropic Messages ↔ Responses around the server's own `responses()`. External harnesses that require an Anthropic endpoint — notably `claude_code_agent` (the Claude Code CLI only talks `/v1/messages`) — can now run against **any** Gym backend (vLLM, OpenAI, an inference provider), not just Anthropic's API. The Messages ↔ Responses mapping lives in one shared, transport-free, SDK-free converter, `nemo_gym/anthropic_converter.py`, used in both directions and shared with the egress `anthropic_model` provider (#1146/#1546). See `DESIGN-NOTE.md` for rationale. # Why Gym standardizes on the Responses API, but real blackbox agents speak other protocols. Rather than a bespoke per-agent bridge or a separate proxy server, the Messages spoke becomes a base-class capability: implement `responses()` once and get `/v1/messages` for free. This is the model-proxy layer the CustomAgent EPIC (#1042) assumes — its sidecar "path-preserving forwards" `/v1/messages` to these routes. # Key design points - **Route on the base class, not a separate proxy.** Every model server inherits `/v1/messages`; no extra process or network hop. (An earlier `anthropic_messages_proxy` + `SimpleMessagesAPIModel` iteration was removed in favor of this.) - **Delegates to the server's own `responses()`** — reuses whatever backend the server has; a small `inspect`-based dispatch handles both existing `responses()` signatures. - **Synthesized SSE.** Backends are non-streaming via Responses; the converter fabricates the Anthropic SSE event stream from the complete response (the Claude CLI requires streaming). - **Shared bidirectional converter**, transport-free and **no `anthropic` SDK** (local shapes over Gym's aiohttp). Fail-loud (`NotImplementedError`) on unsupported content. - **Eval-first.** Token-ids live only at the Responses hop and cannot ride the Anthropic SSE response; an RL path would side-channel them, not thread them through Messages. # Issues Closes #1620 (*"Add a default `/v1/messages` (Anthropic Messages) route to the base Gym model server"*). Closes #1566 Relationship to adjacent work: - **#1620** — this PR is its implementation: the `/v1/messages` route + default Messages ↔ Responses mapping on `SimpleResponsesAPIModel`, the shared converter, and the `claude_code_agent` end-to-end run satisfy its acceptance criteria. - **#1146 / #1546** (egress `anthropic_model`) — inverse direction; shares the same converter. - **#1286** (`responses_converter`) — sibling Chat-Completions ↔ Responses converter; same pattern. - **EPIC #1042** (CustomAgent) — its sidecar model proxy "path-preserving forwards" `/v1/messages` to the routes this PR adds. # Usage Every model server now answers Anthropic Messages directly — this is config-independent and is the core of the change. Against any running Gym model server: ```bash curl -s http://127.0.0.1:<model_server_port>/v1/messages -H 'content-type: application/json' -d '{ "model": "x", "max_tokens": 64, "messages": [{"role": "user", "content": "hi"}] }' ``` `claude_code_agent` reaches any Gym model server by setting its `model_server` ref — the harness resolves `ANTHROPIC_BASE_URL` from it and the CLI appends `/v1/messages` (see `responses_api_agents/claude_code_agent/app.py:_resolve_base_url`). The end-to-end run in **Testing** below used this path against a local vLLM. The verified `reasoning_gym` Claude Code config keeps its direct `anthropic_base_url` default; targeting a Gym model server is an opt-in `model_server` ref. # Additional information - **Changed:** `nemo_gym/base_responses_api_model.py` (new `/v1/messages` route + default `messages()`); **added** `nemo_gym/anthropic_converter.py` (shared bidirectional converter); `responses_api_agents/claude_code_agent/README.md` (documents the model-server path). No change to the verified `reasoning_gym_claude_code_agent.yaml` — targeting a Gym model server is an opt-in `model_server` ref on the agent. - **Converter surface:** `responses_to_anthropic` / `anthropic_to_responses` (egress, shared with #1546); `anthropic_request_to_responses` / `responses_to_anthropic_response` / `anthropic_response_to_sse` (ingress). Supports text, base64 images, function tools, tool_use/tool_result, thinking, stop-reason. Anthropic-API egress *policy* stays in the `anthropic_model` server, not the converter. # Testing - `tests/unit_tests/test_anthropic_converter.py` (ingress + round-trip) and `…_egress.py` (egress) — **100%** converter coverage. - `tests/unit_tests/test_responses_api_model_messages.py` — base-class `/v1/messages` for both `responses()` signatures + SSE framing. - Full unit suite green (284); `openai_model` tests pass unchanged. - **Live e2e:** `claude_code_agent → policy_model (vLLM) /v1/messages` on reasoning_gym — `finished_naturally=1`, 1 turn, ~5.5k tokens, 0 failures (reward reflects the small model's answer, not the plumbing). - Verified: `ruff check`, `ruff format --check`, `pre-commit`. # Open questions - Versioning/regression suite for the converters (OpenAI/Anthropic bump frequently). - Audit/align the existing Chat-Completions ↔ Responses tests. - Confirm fail-loud vs drop policy for items absent in a target schema (e.g. computer-use). --------- Signed-off-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com> Signed-off-by: Lin Jia <linj@nvidia.com> Co-authored-by: Lin Jia <linj@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

ananthsub

some endpoints need to be updated, along with another inline comment, but otherwise this lgtm

…se Gym… (#1627) # What does this PR do? [Design note](#1620 (comment)) Makes **every NeMo Gym model server speak the Anthropic Messages API** by adding a `POST /v1/messages` route to the base `SimpleResponsesAPIModel`, with a default implementation that maps Anthropic Messages ↔ Responses around the server's own `responses()`. External harnesses that require an Anthropic endpoint — notably `claude_code_agent` (the Claude Code CLI only talks `/v1/messages`) — can now run against **any** Gym backend (vLLM, OpenAI, an inference provider), not just Anthropic's API. The Messages ↔ Responses mapping lives in one shared, transport-free, SDK-free converter, `nemo_gym/anthropic_converter.py`, used in both directions and shared with the egress `anthropic_model` provider (#1146/#1546). See `DESIGN-NOTE.md` for rationale. # Why Gym standardizes on the Responses API, but real blackbox agents speak other protocols. Rather than a bespoke per-agent bridge or a separate proxy server, the Messages spoke becomes a base-class capability: implement `responses()` once and get `/v1/messages` for free. This is the model-proxy layer the CustomAgent EPIC (#1042) assumes — its sidecar "path-preserving forwards" `/v1/messages` to these routes. # Key design points - **Route on the base class, not a separate proxy.** Every model server inherits `/v1/messages`; no extra process or network hop. (An earlier `anthropic_messages_proxy` + `SimpleMessagesAPIModel` iteration was removed in favor of this.) - **Delegates to the server's own `responses()`** — reuses whatever backend the server has; a small `inspect`-based dispatch handles both existing `responses()` signatures. - **Synthesized SSE.** Backends are non-streaming via Responses; the converter fabricates the Anthropic SSE event stream from the complete response (the Claude CLI requires streaming). - **Shared bidirectional converter**, transport-free and **no `anthropic` SDK** (local shapes over Gym's aiohttp). Fail-loud (`NotImplementedError`) on unsupported content. - **Eval-first.** Token-ids live only at the Responses hop and cannot ride the Anthropic SSE response; an RL path would side-channel them, not thread them through Messages. # Issues Closes #1620 (*"Add a default `/v1/messages` (Anthropic Messages) route to the base Gym model server"*). Closes #1566 Relationship to adjacent work: - **#1620** — this PR is its implementation: the `/v1/messages` route + default Messages ↔ Responses mapping on `SimpleResponsesAPIModel`, the shared converter, and the `claude_code_agent` end-to-end run satisfy its acceptance criteria. - **#1146 / #1546** (egress `anthropic_model`) — inverse direction; shares the same converter. - **#1286** (`responses_converter`) — sibling Chat-Completions ↔ Responses converter; same pattern. - **EPIC #1042** (CustomAgent) — its sidecar model proxy "path-preserving forwards" `/v1/messages` to the routes this PR adds. # Usage Every model server now answers Anthropic Messages directly — this is config-independent and is the core of the change. Against any running Gym model server: ```bash curl -s http://127.0.0.1:<model_server_port>/v1/messages -H 'content-type: application/json' -d '{ "model": "x", "max_tokens": 64, "messages": [{"role": "user", "content": "hi"}] }' ``` `claude_code_agent` reaches any Gym model server by setting its `model_server` ref — the harness resolves `ANTHROPIC_BASE_URL` from it and the CLI appends `/v1/messages` (see `responses_api_agents/claude_code_agent/app.py:_resolve_base_url`). The end-to-end run in **Testing** below used this path against a local vLLM. The verified `reasoning_gym` Claude Code config keeps its direct `anthropic_base_url` default; targeting a Gym model server is an opt-in `model_server` ref. # Additional information - **Changed:** `nemo_gym/base_responses_api_model.py` (new `/v1/messages` route + default `messages()`); **added** `nemo_gym/anthropic_converter.py` (shared bidirectional converter); `responses_api_agents/claude_code_agent/README.md` (documents the model-server path). No change to the verified `reasoning_gym_claude_code_agent.yaml` — targeting a Gym model server is an opt-in `model_server` ref on the agent. - **Converter surface:** `responses_to_anthropic` / `anthropic_to_responses` (egress, shared with #1546); `anthropic_request_to_responses` / `responses_to_anthropic_response` / `anthropic_response_to_sse` (ingress). Supports text, base64 images, function tools, tool_use/tool_result, thinking, stop-reason. Anthropic-API egress *policy* stays in the `anthropic_model` server, not the converter. # Testing - `tests/unit_tests/test_anthropic_converter.py` (ingress + round-trip) and `…_egress.py` (egress) — **100%** converter coverage. - `tests/unit_tests/test_responses_api_model_messages.py` — base-class `/v1/messages` for both `responses()` signatures + SSE framing. - Full unit suite green (284); `openai_model` tests pass unchanged. - **Live e2e:** `claude_code_agent → policy_model (vLLM) /v1/messages` on reasoning_gym — `finished_naturally=1`, 1 turn, ~5.5k tokens, 0 failures (reward reflects the small model's answer, not the plumbing). - Verified: `ruff check`, `ruff format --check`, `pre-commit`. # Open questions - Versioning/regression suite for the converters (OpenAI/Anthropic bump frequently). - Audit/align the existing Chat-Completions ↔ Responses tests. - Confirm fail-loud vs drop policy for items absent in a target schema (e.g. computer-use). --------- Signed-off-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com> Signed-off-by: Lin Jia <linj@nvidia.com> Co-authored-by: Lin Jia <linj@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Rita Fernandes Neves <rfernandesne@nvidia.com>

Address PR review: `"tool_choice" in body` always evaluated False on the pydantic Body model, forcing tool_choice to "auto" even when callers passed an explicit value. Use `body.tool_choice is not None` in both inference_provider and vllm_model, and add regression tests. Also update Baseten, Nebius, and Friendli base URLs to their current OpenAI-compatible endpoints. Signed-off-by: Chris Wing <cwing@nvidia.com>

…s-model

cwing-nvidia mentioned this pull request May 11, 2026

docs: add inference providers page and improve configure model index #1289

Merged

4 tasks

fix: patch uuid4 at new location in azure_openai_model test

795b018

After extracting ResponsesConverter to nemo_gym/responses_converter.py, the uuid4 monkeypatch must target the new module path. Signed-off-by: Chris Wing <cwing@nvidia.com> Co-authored-by: Cursor <cursoragent@cursor.com>

cwing-nvidia marked this pull request as ready for review May 12, 2026 00:08

cwing-nvidia requested a review from a team as a code owner May 12, 2026 00:08

anwithk requested a review from ananthsub May 22, 2026 21:38

ananthsub previously approved these changes May 29, 2026

View reviewed changes

kajalj22 mentioned this pull request May 29, 2026

ci: smoke test inference provider API keys #1456

Closed

2 tasks

kajalj22 previously approved these changes May 29, 2026

View reviewed changes

refactor: rename chat_completions_model to inference_provider

0b1c642

Names the server by use case (hosted inference providers) rather than the API endpoint shared by all model servers. Signed-off-by: Chris Wing <cwing@nvidia.com>

cwing-nvidia dismissed stale reviews from kajalj22 and ananthsub via 0b1c642 June 1, 2026 06:03

cwing-nvidia changed the title ~~feat: add chat_completions_model server for hosted inference providers~~ feat: add inference_provider server for hosted inference providers Jun 1, 2026

merge: resolve main into cwing/chat-completions-model

67f7736

Keep both coverage omit entries: responses_converter.py (PR) and environments/*/prepare.py (main). Signed-off-by: Chris Wing <cwing@nvidia.com>

copy-pr-bot Bot temporarily deployed to public June 1, 2026 06:27 Inactive

copy-pr-bot Bot temporarily deployed to public June 1, 2026 06:29 Inactive

fix: restore Field import in vllm_model after merge

11334a8

Merge with main dropped the pydantic Field import while keeping default_headers on VLLMModelConfig. Signed-off-by: Chris Wing <cwing@nvidia.com>

copy-pr-bot Bot temporarily deployed to public June 1, 2026 06:29 Inactive

copy-pr-bot Bot temporarily deployed to public June 1, 2026 06:30 Inactive

copy-pr-bot Bot temporarily deployed to public June 1, 2026 06:31 Inactive

anwithk added this to NeMo Gym 0.4.0 - July 1 Jun 3, 2026

github-project-automation Bot moved this to Dev Todo in NeMo Gym 0.4.0 - July 1 Jun 3, 2026

kajalj22 self-requested a review June 5, 2026 20:41

kajalj22 previously approved these changes Jun 5, 2026

View reviewed changes

copy-pr-bot Bot temporarily deployed to public June 16, 2026 17:40 Inactive

copy-pr-bot Bot temporarily deployed to public June 16, 2026 17:41 Inactive

copy-pr-bot Bot temporarily deployed to public June 16, 2026 17:42 Inactive

ffrujeri mentioned this pull request Jun 16, 2026

feat: Add a default /v1/messages (Anthropic Messages) route to the base Gym… #1627

Merged

Merge branch 'main' into cwing/chat-completions-model

33fb6f4

Signed-off-by: Lin Jia <linj@nvidia.com> # Conflicts: # fern/versions/latest/pages/model-server/vllm.mdx

This was referenced Jun 22, 2026

refactor: share concurrency-limiting semaphore across model servers #1655

Open

proposal: reconsider the policy_ naming convention for model server config/env variables #1658

Open

cwing-nvidia linked an issue Jun 22, 2026 that may be closed by this pull request

docs: configure models - openai and azure openai #665

Closed

1 task

cwing-nvidia requested review from ananthsub and cmunley1 June 22, 2026 22:35

anwithk moved this from Dev Todo to Review WIP in NeMo Gym 0.4.0 - July 1 Jun 25, 2026

ananthsub previously approved these changes Jun 25, 2026

View reviewed changes

cwing-nvidia mentioned this pull request Jun 26, 2026

Provider-aware endpoint error handling in inference_provider model server #1748

Open

cwing-nvidia dismissed ananthsub’s stale review via 618d8cc June 26, 2026 04:04

Merge remote-tracking branch 'origin/main' into cwing/chat-completion…

7a8c99b

…s-model

kajalj22 self-requested a review June 26, 2026 04:43

kajalj22 approved these changes Jun 26, 2026

View reviewed changes

cwing-nvidia merged commit d6d7d97 into main Jun 26, 2026
17 checks passed

github-project-automation Bot moved this from Review WIP to Done in NeMo Gym 0.4.0 - July 1 Jun 26, 2026

cwing-nvidia deleted the cwing/chat-completions-model branch June 26, 2026 05:21

This was referenced Sep 15, 2026

Document a vendor-neutral model-routing evaluation workflow #3298

Open

Route-aware evaluation for user-provided routers and gateways #3416

Open

Navigation Menu

Repository navigation

feat: add inference_provider server for hosted inference providers - #1286

feat: add inference_provider server for hosted inference providers#1286
cwing-nvidia merged 16 commits into
mainfrom
cwing/chat-completions-model

cwing-nvidia commented May 10, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented May 10, 2026

Uh oh!

cmunley1 commented May 29, 2026

Uh oh!

cmunley1 commented May 29, 2026

Uh oh!

cwing-nvidia commented Jun 1, 2026

Uh oh!

github-actions Bot commented Jun 16, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Jun 22, 2026

Uh oh!

ananthsub left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

cwing-nvidia commented May 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Motivation

Note

Test plan

Uh oh!

copy-pr-bot Bot commented May 10, 2026

Uh oh!

cmunley1 commented May 29, 2026

Uh oh!

cmunley1 commented May 29, 2026

Uh oh!

cwing-nvidia commented Jun 1, 2026

Uh oh!

github-actions Bot commented Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

copy-pr-bot Bot commented Jun 22, 2026

Uh oh!

ananthsub left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

cwing-nvidia commented May 10, 2026 •

edited

Loading

github-actions Bot commented Jun 16, 2026 •

edited

Loading