Repository navigation
Super 3.5 vllm model tuning (Qwen3.5-122B-A10B, Inkling-Small) - #2599
Merged
Merged
Conversation
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
Disable prefix caching, reduce the maximum sequence count from 512 to 256, match CUDA-graph capture to 256, and enforce eager execution to isolate the decode stalls observed during full-suite evaluation. Signed-off-by: Frankie Siino <fsiino@nvidia.com>
Pass num_samples_in_parallel only when explicitly configured. Validate the value and filter stale exclusions against Slurm's current node inventory before submission. Signed-off-by: Frankie Siino <fsiino@nvidia.com>
Remove FP8 KV caching and eager execution while retaining the 256 sequence and capture limits. Bound each request to four images and one video. Signed-off-by: Frankie Siino <fsiino@nvidia.com>
Store attention KV blocks in FP8 to reduce cache memory traffic and preserve
decode throughput. Disable runtime scale calculation because Qwen3.5 hybrid-cache
warmup produced invalid inferred scales.
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
Allow each decode node to run a local TP/EP replica instead of forming a cross-
node DP/EP group. Register every replica with the PD router and add an opt-in
decode policy for balanced variable-length eval traffic. Preserve the existing
coupled and consistent-hash defaults.
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
Parse combined Slurm logs from coupled or independent decode deployments and
report synchronized aggregate throughput, output tokens per second per request,
waiting requests, and GPU KV-cache usage.
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
Sandbox pwd output included a trailing newline, causing the agent to request /testbed\n/export.json and receive FILE_NOT_FOUND. Strip the output before building the path and test that /testbed/export.json is
downloaded.
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
…odel-tuning Signed-off-by: Frankie Siino <fsiino@nvidia.com> # Conflicts: # benchmarks/nemotron_3.5_super/build-super-vl-rl-v0251-thin.sh # benchmarks/nemotron_3.5_super/build_eval_container.sh # benchmarks/nemotron_3.5_super/sbatch_external_vllm.sh
Use a stable output path and enable Gym's rollout cache when RESUME_EVAL_ON_REQUEUE=1. This prevents requeued jobs from losing completed work by starting under a new timestamped directory, while keeping the existing behavior unchanged by default. Signed-off-by: Frankie Siino <fsiino@nvidia.com>
Move topology and batching arguments into the prefill and decode configs, raise max sequences to 512, and optionally enable MTP on both roles. Signed-off-by: Frankie Siino <fsiino@nvidia.com>
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
…odel-tuning Signed-off-by: Frankie Siino <fsiino@nvidia.com>
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
fsiino-nvidia
marked this pull request as ready for review
August 20, 2026 16:45
fsiino-nvidia
changed the title
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
…odel-tuning Signed-off-by: Frankie Siino <fsiino@nvidia.com> # Conflicts: # benchmarks/nemotron_3.5_super/sbatch_external_vllm.sh
junkeun-nexusflow
suggested changes
Aug 20, 2026
Default to zero retries after a malformed tool response and expose malformed_tool_call_max_retries for explicit overrides. Signed-off-by: Frankie Siino <fsiino@nvidia.com>
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
Merged
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
github-actions
Bot
added
the
sla:review-overdue
jkyi-nvidia
previously approved these changes
Aug 24, 2026
…/super35-vllm-model-tuning Signed-off-by: Brian Yu <bxyu@nvidia.com>
bxyu-nvidia
approved these changes
Aug 26, 2026
Contributor
|
/ok to test 4eb599a |
jkyi-nvidia
enabled auto-merge (squash)
August 26, 2026 17:11
agronskiy
added a commit
that referenced
this pull request
Sep 1, 2026
Merges `main` at b793621 ("Super 3.5 vllm model tuning (Qwen3.5-122B-A10B, Inkling-Small)", #2599) rather than the current `main` tip. WHY THIS TARGET b793621 is the commit immediately before 65129dd ("chore(deps): pin openai to 2.44.0", #2456). That PR added class NeMoGymChatCompletionCreateParamsNonStreaming(BaseModel): model_config = ConfigDict(extra="forbid") which makes the proxy reject every agentic chat-completions request that carries `chat_template_kwargs` — the field agentic GDPVal runs use to drive thinking budgets on self-hosted models. Each such request comes back as 422 Unprocessable Entity {"detail":[{"type":"extra_forbidden","loc":["body","chat_template_kwargs"], "msg":"Extra inputs are not permitted","input":{"thinking":true}}]} so the rollout fails before it reaches the model. Merging `main` past 65129dd would take that regression into the GDPVal branch and break the production runs this branch exists to serve. b793621 was chosen because it is the last commit that is regression-free while still giving the branch everything else it needs: * `requires-python = ">=3.13.14"`, so the py313 runtime container stays correct; * `extra="forbid"` appears exactly once in nemo_gym/openai_utils.py, on `NeMoGymResponseCreateParamsNonStreaming` (the pre-existing 2025 Responses-class validation) and NOT on the ChatCompletion class; * the rollout-observability work (bc521f7) is an ancestor. CONFLICTS AND RESOLUTIONS Eight files conflicted; all GDPVal files auto-merged. Every conflict was resolved to keep both sides' behaviour rather than pick a winner. * nemo_gym/cli/env.py, cli/setup_command.py and their tests: `get_venv_path` and `resolve_server_venv_path` are the same function under two names (byte-identical bodies). Canonicalised on devel's name and docstring, kept main's `Path(...)` normalisation of `root_venv_path`, renamed main's call sites and the `monkeypatch.setattr` in test_cli.py, and dropped the now-unused `ROOT_DIR` import. * nemo_gym/openai_utils.py: took main's content-part types — its `video_url` part accepts `Union[str, Dict[str, Any]]` where devel's was dict-only, and it adds `NeMoGymChatCompletionContentPartFileParam` — while keeping devel's comment recording the 422 that motivated adding a video part at all. * responses_api_models/local_vllm_model/setup.py: both sides pin vllm==0.24.0 at this merge target, so the resolution is main's file (which additionally pins flashinfer-python==0.6.12) plus devel's note that 0.24.0 is the first release with MiniMax-M3 support. * nemo_gym/rollout_reverification.py: plain union of both import lists from nemo_gym.rollout_collection; dropped devel's `NG_TERMINAL_KEY as NG_TERMINAL_KEY` re-export idiom, nothing needs it. * nemo_gym/rollout_collection.py: kept both sides' features — main's exporters (`upload_rollouts`/`export_rollouts`/`get_exporters`, replacing the W&B-specific path) and token-capture retirement, devel's dispatch budget, drain margin and `DispatchLatencyTracker`, `kill_shaped` no-persist rows, `_validate_dispatch_concurrency` and `ordered_tasks`. Three resolutions here are semantic rather than textual: - main's `from time import time` SHADOWS the `time` module that devel's `time.monotonic()` calls need, so the module import was kept and main's two bare `time()` calls rewritten as `time.time()`; taking main's side verbatim compiles and then fails at runtime; - the persistence branch takes main's flat `if no_persist / elif failure_class is not None / else` chain with its token-capture retirement, dropping devel's `result_strs.append` (it fed the removed W&B table) and the stale "not the W&B table either" wording in the kill_shaped comment; - devel's `ordered_tasks = [asyncio.ensure_future(...)]` scheduling was kept, because main still passes bare coroutines to `as_completed`, which makes `dispatch_longest_first` a no-op, and the shared code after the conflict references `ordered_tasks`. * tests/unit_tests/test_rollout_collection.py: import union, and devel's version of the fresh-run cleanup test, which is a superset of main's and additionally asserts the aggregate-metrics file is cleared. The fresh-run cleanup in `run_from_config` now unlinks the failures sidecar and the aggregate-metrics file so a re-run cannot inherit retry attempts or published metrics from an older run at the same output path. VERIFICATION `ruff check` and `ruff format --check` with the CI-pinned ruff 0.9.9 are clean across nemo_gym, tests, responses_api_models and resources_servers. resources_servers/gdpval/tests: 386 passed, 11 skipped — the `reference_missing` path (#2796) and `strict_comparison_trials` (#2807) are both intact. tests/unit_tests against an `origin/main` baseline built in a scratch worktree on the same venv: 79 failures on the baseline, 101 here. The 44 new failures are all in test_openai_utils.py (43) plus test_responses_api_model_streaming.py::test_prunes_nested_extra_fields, and are an environment artefact, not a merge defect: b793621 predates #2456 and still declares `openai<=2.7.2`, while the venv has openai 2.44.0 installed. Those tests enumerate the *installed* SDK's item tags (`shell_call`, `apply_patch_call`, `compaction`, `tool_search_call`, ...) and require a Gym union member for each; the pre-2.44 schemas in this tree have none. Re-running the same suite with an openai inside the declared pin (2.7.2 shadowed onto PYTHONPATH, venv untouched) gives 1 failed / 2976 passed — the single failure, test_opensandbox_cleanup.py::test_script_help_runs_by_ direct_path, is also present in the origin/main baseline. So the new-failure set against baseline is EMPTY once the SDK matches the pin, and no failure touches rollout collection, reverification, the CLI or GDPVal. Signed-off-by: Alex Gronskiy <agronskiy@nvidia.com>
khazic
pushed a commit
to khazic/Gym-lao
that referenced
this pull request
Sep 19, 2026
_(Originally branched from `fsiino/super35-vllm-model-tuning`, PR NVIDIA-NeMo#2599. After that PR merged, we moved the Ultra changes onto updated main and kept the shared launcher compatible. Main was subsequently merged again at `f656822f6`.)_ ## What this PR does - Adds `benchmarks/nemotron_3.5_super/vllm_configs/nemotron_3_ultra.sh`: a fixed Ultra BF16 recipe using piecewise CUDA graphs on both tiers, MTP3, prefix caching, asynchronous prefill, synchronous decode, FP8 KV cache, FP16 Mamba state with stochastic rounding, and expert parallelism. - Adds opt-in coupled deployment to `benchmarks/nemotron_3.5_super/sbatch_external_vllm.sh`. Ultra uses four prefill nodes and four decode nodes, each with four GPUs; each tier distributes the model across 16 GPUs. The default remains independent deployment for models that fit a complete replica on one node. Existing router defaults and sampling overrides are preserved. - Makes walltime and allocation segmentation configurable, bounds coupled startup health probes, and detects coupled prefill/router exits during startup and serving. Adds launcher regression coverage in `tests/unit_tests/test_super_vllm_launcher.py`. ## Benchmark scores | Benchmark | Earlier full run `7014044 / f656822` | Non-eager full run `7093754` | Tracker | |---|---:|---:|---:| | Tau3-Banking | 11.13% | 11.34% | 18.40% | | Tau3-Average | 58.86% | 58.13% | 64.60% | | SciCode — full-problem accuracy | 11.54% | 9.62% | 11.90% | | AA-LCR | 52.13% | 54.22%* | 64.10% | | AA-Omniscience — OmniIndex | 7.65% | 7.58% | 23.80% | | GPQA Diamond | 85.98% | 87.56% | 86.50% | *AA-LCR's newer score covers 1,457/1,600 attempts, versus 1,600/1,600 previously; the apparent increase is not a clean comparison. Tau3-Average equally weights Airline, Retail, Telecom, and Banking. Omniscience answer accuracy was 24.38%/24.08%; that is a different metric from OmniIndex and must not be compared with the tracker's OmniIndex value. Score artifacts: - `results/nemotron-3-ultra-bf16-p4d4-c16-mtp3-v0271-piecewise-copyinputs-full-suite-validation-20260908/resumable_aggregate_metrics.json` - `results/nemotron-3-ultra-mtp3-c16-syncdecode-full-super-20260912-023329/resumable_aggregate_metrics.json` ## Run command — Super suite, post-merge 0.29 (30 minutes) ```bash EXPERIMENT_NAME="nemotron-3-ultra-mtp3-c24-v0290-postmerge-$(date -u +%Y%m%d-%H%M%S)" MODEL=/lustre/fsw/portfolios/llmservice/users/igitman/hf_models/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 \ VLLM_CONFIG=benchmarks/nemotron_3.5_super/vllm_configs/nemotron_3_ultra.sh \ EXPERIMENT_NAME="$EXPERIMENT_NAME" \ ROLLOUTS_FPATH="results/$EXPERIMENT_NAME/resumable.jsonl" \ EXPORT_TO_CSV=0 \ NUM_PREFILL_NODES=4 \ NUM_DECODE_NODES=4 \ VLLM_PD_DEPLOYMENT_MODE=coupled \ VLLM_SLURM_SEGMENT=4 \ SBATCH_ACCOUNT=nemotron_rl_algo \ SBATCH_PARTITION=batch_long \ SBATCH_QOS=normal \ SBATCH_TIME=2-00:00:00 \ SBATCH_GRES=gpu:4 \ SBATCH_EXPORT=ALL \ CONTAINER="/lustre/fsw/portfolios/llmservice/users/fsiino/repos/Gym-inkling-tuning/results/vllm/vllm-openai:v0.29.0___super-vl-evals-2a3c65b-ultra-with-gym-b894cbe-20260914-202306.sqsh" \ MOUNTS="/lustre:/lustre,$(pwd -P)/env.yaml:/opt/Gym/env.yaml:x-create=file,$(pwd -P)/results:/opt/Gym/results,$(pwd -P)/benchmarks/nemotron_3.5_super/vllm_configs/nemotron_3_ultra.sh:/opt/Gym/benchmarks/nemotron_3.5_super/vllm_configs/nemotron_3_ultra.sh:ro" \ bash benchmarks/nemotron_3.5_super/sbatch_external_vllm.sh \ --config benchmarks/nemotron_3.5_super/eval_container_config.yaml \ '++num_samples_in_parallel=24' \ '++resume_from_cache=true' \ '++critpt_benchmark_agent.responses_api_agents.critpt_agent.datasets=[]' \ '++hle_benchmark_equivalence_llm_judge_simple_agent.responses_api_agents.simple_agent.datasets=[]' ``` The two dataset-list overrides exclude CRITPT and HLE. Remove the corresponding final argument to include either benchmark in a **fresh** experiment. `EXPORT_TO_CSV=0` skips the optional exporter, whose Omniscience metric-key mismatch remains unresolved; evaluation JSON and final aggregate metrics are still written. ## Run command — SWE-bench Verified, post-merge 0.29 (30 minutes) Uses the same Ultra recipe, container, concurrency 24, and coupled P4/D4 allocation as Super, with the OpenCode agent and OpenSandbox configs. Run from the repository root with the existing OpenSandbox connection configured in `env.yaml` or the environment. The distinct experiment name keeps SWE-bench results separate; no CRITPT/HLE exclusions are needed because the Super config is not loaded. ```bash EXPERIMENT_NAME="nemotron-3-ultra-mtp3-c24-v0290-postmerge-swebench-verified-$(date -u +%Y%m%d-%H%M%S)" MODEL=/lustre/fsw/portfolios/llmservice/users/igitman/hf_models/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 \ VLLM_CONFIG=benchmarks/nemotron_3.5_super/vllm_configs/nemotron_3_ultra.sh \ EXPERIMENT_NAME="$EXPERIMENT_NAME" \ ROLLOUTS_FPATH="results/$EXPERIMENT_NAME/resumable.jsonl" \ EXPORT_TO_CSV=0 \ NUM_PREFILL_NODES=4 \ NUM_DECODE_NODES=4 \ VLLM_PD_DEPLOYMENT_MODE=coupled \ VLLM_SLURM_SEGMENT=4 \ SBATCH_ACCOUNT=nemotron_rl_algo \ SBATCH_PARTITION=batch_long \ SBATCH_QOS=normal \ SBATCH_TIME=2-00:00:00 \ SBATCH_GRES=gpu:4 \ SBATCH_EXPORT=ALL \ CONTAINER="/lustre/fsw/portfolios/llmservice/users/fsiino/repos/Gym-inkling-tuning/results/vllm/vllm-openai:v0.29.0___super-vl-evals-2a3c65b-ultra-with-gym-b894cbe-20260914-202306.sqsh" \ MOUNTS="/lustre:/lustre,$(pwd -P)/env.yaml:/opt/Gym/env.yaml:x-create=file,$(pwd -P)/results:/opt/Gym/results,$(pwd -P)/benchmarks/nemotron_3.5_super/vllm_configs/nemotron_3_ultra.sh:/opt/Gym/benchmarks/nemotron_3.5_super/vllm_configs/nemotron_3_ultra.sh:ro" \ bash benchmarks/nemotron_3.5_super/sbatch_external_vllm.sh \ --config responses_api_models/vllm_model/configs/vllm_model.yaml \ --config benchmarks/swebench/verified/opencode.yaml \ --config nemo_gym/sandbox/providers/opensandbox/configs/opensandbox.yaml \ '++num_samples_in_parallel=24' \ '++resume_from_cache=true' ``` Each command reserves eight nodes / 32 GPUs; running both simultaneously requires 16 nodes / 64 GPUs. These are separate 30-minute validation jobs, not full-suite score runs. Check model/router readiness, live generation, tokens/sec/request, waiting, KV usage, and saved-output health; additionally check sandbox startup and agent activity for SWE-bench. Startup and sandbox preparation consume part of the allocation, so SWE-bench may save few completed attempts within this window. The launcher submits a separate job-scoped cleanup job for each evaluation. ## Command controls and resume | Control | Valid values | Why this command uses it | |---|---|---| | `VLLM_PD_DEPLOYMENT_MODE` | `independent` (default), `coupled` | Branch-added opt-in for multi-node tiers; Ultra requires `coupled` because a complete replica does not fit on one four-GPU node. | | `VLLM_SLURM_SEGMENT` | Unset or a positive integer | Branch-added control; overrides main's `SEGMENT`. If both are unset/empty, use all allocated nodes as one segment. `4` aligns each segment with a four-node tier, the placement used in successful Ultra runs. | | `SEGMENT` | Unset or a positive integer | Main's equivalent segmentation control, honored when `VLLM_SLURM_SEGMENT` is unset/empty. No need to set both. | | `SBATCH_TIME` | Slurm duration within account/partition limits | Branch makes the prior `04:00:00` default configurable. `00:30:00` bounds this tuning test; it is not enough for the full suite. | | `VLLM_USE_V2_MODEL_RUNNER` | Launcher now explicitly sets `0` | Main selects V1, which was faster in our tests. This no longer needs a command argument; submitting `1` will be overridden by the launcher. V1 is version-specific and deprecated, so future containers need revalidation. | | `SBATCH_EXPORT` | Slurm export setting; `ALL` here | Existing Slurm control; carries the run environment, including the fixed output path, into the allocation. | | `++num_samples_in_parallel` | Positive integer | Existing Gym concurrency control. `24` improved aggregate generation relative to `20` while averaging above 90 tokens/sec/request; it does not prove proportionally faster suite completion. | | `ROLLOUTS_FPATH` + `++resume_from_cache` | Fixed output JSONL path; `true` or `false` | Existing controls. Set a unique mounted results path and `true` on the first submission to preserve completed work across Slurm requeues. | For a **manual resume**, wait until the previous job stops, replace the timestamp-generating `EXPERIMENT_NAME` assignment with its exact previous experiment name, and rerun with the same fixed output path, model, recipe, container, datasets, and resume override. Preserve both `resumable.jsonl` and `resumable_materialized_inputs.jsonl`; the latter records the selected tasks. A Slurm requeue reuses the submitted environment without needing that manual edit. Interrupted, unsaved work may be repeated; neither control configures Slurm's requeue policy. Without `ROLLOUTS_FPATH`, the launcher chooses a new timestamped output path at each restart, so `++resume_from_cache=true` alone will not find the prior results. Without resume enabled, Gym may clear the selected output. For any new tuning configuration or dataset selection, use a fresh experiment/output path rather than mixing results with a previous run. ## Changes since review at `f656822f6` - **Hardcoded the tuned settings:** removed `ULTRA_*` switches and their validation branches. Prefill/decode memory utilization is `0.90/0.95`, batched-token limits are `16384/8192`, maximum sequences is `64`, and MTP drafts three extra tokens. Graph modes, copied inputs, and decode capture sizes are fixed in the recipe. - **Removed eager enforcement:** both tiers use `PIECEWISE` graphs with copied inputs; no `--enforce-eager` flag or eager fallback remains. - **Enabled prefix caching:** removed the old disabling flag and warning. The original multiple-local-block assertion did not recur in the subsequent full vLLM 0.27.1 run or the short 0.29 tests. This is not a claim that all cache warnings or output-quality problems are fixed. - **Applied async scheduling where supported by tuning:** enabled on prefill, disabled on decode. Decode async was retested on 0.29 in `7153011`; matching post-warmup durations gave 89.96 versus 95.55 tokens/sec/request with synchronous decode, approximately 6% slower. The reviewer recommendation was to use it if it helps, not require it on both tiers. - **Reused existing concurrency/resume controls:** removed `NUM_SAMPLES_IN_PARALLEL` and `RESUME_EVAL_ON_REQUEUE`. Commands now use Hydra's `++num_samples_in_parallel`, a fixed `ROLLOUTS_FPATH`, and `++resume_from_cache=true`. Launcher tests cover argument preservation and stable output paths across requeues and manual restarts. - **Reconciled the latest main launcher changes:** preserved main's V1 runner setting, router-default balancing thresholds, and `cpu-normal` cleanup QOS. Support both main's `SEGMENT` and our higher-priority `VLLM_SLURM_SEGMENT`, falling back to the calculated node count and emitting exactly one `--segment` argument. Coupled serving remains opt-in. --------- Signed-off-by: Frankie Siino <fsiino@nvidia.com>
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds model-specific serving configurations for Qwen3.5-122B-A10B and Inkling Small, configurable decode routing and evaluation concurrency, resumable rollouts, fail-fast multi-node serving, and benchmark-level failure containment.
Shared infrastructure and benchmark reliability
cache_awaredecode-routing policy while preserving that default for callers that do not set an override.RESUME_EVAL_ON_REQUEUE=1 \ EXPERIMENT_NAME="<stable-experiment-name>" \ bash benchmarks/nemotron_3.5_super/sbatch_external_vllm.sh ...--kill-on-bad-exit=1so one failed vLLM worker terminates the server step and propagates through the launcher's existing lifecycle monitoring instead of leaving a partial deployment running. The Gym readiness timeout remains configurable.gym eval prepareand verify it while building the evaluation image.pwdnewline before building the export path, preventing valid/testbed/export.jsonartifacts from being requested with a trailing newline.Qwen3.5-122B-A10B
Model-specific changes
QWEN_ENABLE_MTP=1; matching settings preserve compatible transferred KV-cache layouts.benchmarks/nemotron_3.5_super/vllm_configs/qwen3.5-122b-a10b.sh.How to run
Validation
The full 11,213-rollout suite completed on HSG GB200 with P1/D4, concurrency 64, TP4/EP4, MTP2, and zero failed rollouts.
Inkling Small
Model-specific changes
benchmarks/nemotron_3.5_super/vllm_configs/inkling_small.sh.Container
Inkling required a separate recipe-compatible image layered with Gym and
vllm-router. The validated image isresults/vllm/vllm-openai:nightly-inkling-small-20260815_with_gym.sqsh, built from vLLM0.27.2rc1.dev77+gac7509e2b. The recipe's compatibility floor is vLLM 0.26; this records the exact newer build tested.How to run
Note: The full suite also requires the externally staged, gitignored
benchmarks/scicode/data/test_data.h5fixture.Validation
The validation run completed all 11,213 rows on HSG GB200 in 2:30:13. It produced 11,212 normal rows and one structured failure caused by an external Omniscience judge HTTP 500, not by Inkling or vLLM.
54cee4f1(merge commitef1dd91b) enabled CRITPT in the Super suite. CRITPT's public scorer waits for 70 distinct problems before scoring a batch. The suite interleaves 5x70-item batches. At concurrency 64 (our tuned setting for optimal output tokens/s/request) every Gym slot can become occupied by a rollout waiting for an incomplete batch and block eval progress.To clear CRITPT, you can cancel and resubmit with
RESUME_EVAL_ON_REQUEUE=1andNUM_SAMPLES_IN_PARALLEL=350. This will lower output tokens/s/request but provide enough slots for all 350 CRITPT rollouts to complete their batches. Once all 350 CRITPT rows have been scored and written toresumable.jsonl, either cancel and resume at concurrency 64 or allow the run to finish at concurrency 350.