Repository navigation
Conversation
|
Pushed SGLang hard-rejects input The splice fix + this guard have now been exercised end-to-end by a 16-node R=32 SGLang async-GRPO run: 0 chat-500s, 15+ training steps with graceful trajectory truncation and no stall. Still draft pending a full convergence-vs-baseline comparison. |
|
Convergence validation complete. The splice fix + over-context guard in this PR (as carried on the fork branch) have now been validated by a full-scale end-to-end run: Qwen3-30B-A3B-Thinking SWE-bench async-GRPO on SGLang, 16 nodes (8 train + 8 gen, 32 generation replicas), 63 training steps across four clean checkpoint+resume segments.
Remaining known caveat: this port of the fix onto refactored Gym main is still not GPU-tested in this exact form (the validated code is the fork's pre-refactor equivalent) — keeping the PR draft until that's run. |
Signed-off-by: Serge Panev <spanev@nvidia.com>
SGLang /generate hard-rejects input tokens >= context_length (HTTP 500), unlike vLLM which tolerates it. In a multi-turn SWE rollout the spliced prompt can already fill the window; the 500 then corrupts the trajectory and trips the downstream contiguity assert, stalling the whole async step. Mirror the vLLM context-length handling: pre-check the prompt length against sglang_max_total_sequence_length, and defensively catch the ClientResponseError, ending the turn cleanly with finish_reason="length" (empty completion) in both cases. Validated by a 16-node R=32 SGLang GRPO run (0 chat-500s; 15+ steps, graceful truncation, no stall). Signed-off-by: Serge Panev <spanev@nvidia.com>
The SGLang engine path re-parses tool calls from raw generated text client-side. The existing parser only handled the hermes JSON format (Qwen3-thinking). Add a dependency-free parser for the qwen3_coder XML-ish format (<tool_call><function=NAME><parameter=KEY>...) used by e.g. Qwen3-Coder-style chat templates, with schema-aware argument type coercion from the request's tools, selectable via a new sglang_tool_format config field (default "hermes" = no behavior change). Unit tests mirror the template's canonical example. Signed-off-by: Serge Panev <spanev@nvidia.com>
Chat templates iterate assistant tool-call arguments as a mapping
(tool_call.arguments|items), but OpenAI-format history carries
function.arguments as a JSON string - exactly what our /generate-path
parsers emit. When a request misses the token-splice cache and falls
back to a full template render of a tool-call-bearing conversation,
jinja raises TypeError('Can only get item pairs from a mapping.') and
the request 500s.
Normalize on the full-render path only: decode string arguments that
parse to a JSON object, leave everything else (and the splice path,
which never re-renders old assistant turns) untouched. Inputs are not
mutated - the splice session cache holds the original messages.
Signed-off-by: Serge Panev <spanev@nvidia.com>
Signed-off-by: Serge Panev <spanev@nvidia.com>
Mechanical ruff check --fix (I001 import order) + ruff format on the SGLang proxy files; no logic changes. Fixes the failing Lint check on the PR. Signed-off-by: Serge Panev <spanev@nvidia.com>
Signed-off-by: Serge Panev <spanev@nvidia.com>
Signed-off-by: Serge Panev <spanev@nvidia.com>
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
Kh4L
changed the title
The adapter renders the chat template and tokenizes locally before calling the SGLang server, so its transformers version has to agree with the server's. It was pinned to 5.6.0 to match the NeMo-RL sglang extra; that extra moves to 5.8.1 alongside sglang 0.5.13.post1, so this pin follows. A mismatch here does not fail loudly at import. It surfaces as a rollout contiguity failure when the locally rendered prefix stops matching what the server tokenized, which is far harder to attribute, hence the comment on the pin. Signed-off-by: Serge Panev <spanev@nvidia.com>
svcnvidia-nemo-ci
added
area:model
Signed-off-by: Serge Panev <spanev@nvidia.com>
|
/ok to test 3122202 |
What changes
Adds a text-only SGLang model server for NeMo-Gym RL rollouts. The adapter uses
/generateto retain exact sampled token IDs and logprobs, and splices those sampled IDs into later turns so the trainer sees a contiguous token prefix.The adapter also preserves reasoning delimiters, parses Hermes/Qwen tool calls, rejects changed rendering inputs within a cached session, and ends a turn cleanly when its prompt fills the context window.
Current-main compatibility
1e5a12e27c4a60a3ccca3139acb18465953aac7c.create_generateintonemo_gym.client.openai_client, using the current shared request, status and JSON-reading paths. The removed flat client module stays removed.5.12.1, matching NeMo-RL's SGLang extra.The final comparison against main contains 13 files. The root dependency manifest/lock and the existing vLLM adapter match main.
Validation
create_generate.pre-commit run --all-filespassed on the identical merged source tree with the PR additions staged against current main, including formatting, imports, package layering and the guardword check.git diff --checkpassed; commit is signed off.This remains a draft pending a representative model-backed SGLang rollout on the refreshed head. Earlier fork training runs do not validate this port's current GPU path.
Related work
Supports NVIDIA-NeMo/RL#3127. Episode truncation is tracked separately in NVIDIA-NeMo/RL#3063; single-controller and multimodal integration remain separate work. No additional issue is needed for this existing SGLang integration PR.