Ultra 3 vllm tuning by fsiino-nvidia · Pull Request #2645 · NVIDIA-NeMo/Gym

fsiino-nvidia · 2026-08-21T03:26:32Z

(Originally branched from fsiino/super35-vllm-model-tuning, PR #2599. After that PR merged, we moved the Ultra changes onto updated main and kept the shared launcher compatible. Main was subsequently merged again at f656822f6.)

What this PR does

Adds benchmarks/nemotron_3.5_super/vllm_configs/nemotron_3_ultra.sh: a fixed Ultra BF16 recipe using piecewise CUDA graphs on both tiers, MTP3, prefix caching, asynchronous prefill, synchronous decode, FP8 KV cache, FP16 Mamba state with stochastic rounding, and expert parallelism.
Adds opt-in coupled deployment to benchmarks/nemotron_3.5_super/sbatch_external_vllm.sh. Ultra uses four prefill nodes and four decode nodes, each with four GPUs; each tier distributes the model across 16 GPUs. The default remains independent deployment for models that fit a complete replica on one node. Existing router defaults and sampling overrides are preserved.
Makes walltime and allocation segmentation configurable, bounds coupled startup health probes, and detects coupled prefill/router exits during startup and serving. Adds launcher regression coverage in tests/unit_tests/test_super_vllm_launcher.py.

Benchmark scores

Benchmark	Earlier full run `7014044 / f656822f6`	Non-eager full run `7093754`	Tracker
Tau3-Banking	11.13%	11.34%	18.40%
Tau3-Average	58.86%	58.13%	64.60%
SciCode — full-problem accuracy	11.54%	9.62%	11.90%
AA-LCR	52.13%	54.22%*	64.10%
AA-Omniscience — OmniIndex	7.65%	7.58%	23.80%
GPQA Diamond	85.98%	87.56%	86.50%

*AA-LCR's newer score covers 1,457/1,600 attempts, versus 1,600/1,600 previously; the apparent increase is not a clean comparison. Tau3-Average equally weights Airline, Retail, Telecom, and Banking. Omniscience answer accuracy was 24.38%/24.08%; that is a different metric from OmniIndex and must not be compared with the tracker's OmniIndex value.

Score artifacts:

results/nemotron-3-ultra-bf16-p4d4-c16-mtp3-v0271-piecewise-copyinputs-full-suite-validation-20260908/resumable_aggregate_metrics.json
results/nemotron-3-ultra-mtp3-c16-syncdecode-full-super-20260912-023329/resumable_aggregate_metrics.json

Run command — Super suite, post-merge 0.29 (30 minutes)

EXPERIMENT_NAME="nemotron-3-ultra-mtp3-c24-v0290-postmerge-$(date -u +%Y%m%d-%H%M%S)"

MODEL=/lustre/fsw/portfolios/llmservice/users/igitman/hf_models/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 \
VLLM_CONFIG=benchmarks/nemotron_3.5_super/vllm_configs/nemotron_3_ultra.sh \
EXPERIMENT_NAME="$EXPERIMENT_NAME" \
ROLLOUTS_FPATH="results/$EXPERIMENT_NAME/resumable.jsonl" \
EXPORT_TO_CSV=0 \
NUM_PREFILL_NODES=4 \
NUM_DECODE_NODES=4 \
VLLM_PD_DEPLOYMENT_MODE=coupled \
VLLM_SLURM_SEGMENT=4 \
SBATCH_ACCOUNT=nemotron_rl_algo \
SBATCH_PARTITION=batch_long \
SBATCH_QOS=normal \
SBATCH_TIME=2-00:00:00 \
SBATCH_GRES=gpu:4 \
SBATCH_EXPORT=ALL \
CONTAINER="/lustre/fsw/portfolios/llmservice/users/fsiino/repos/Gym-inkling-tuning/results/vllm/vllm-openai:v0.29.0___super-vl-evals-2a3c65b-ultra-with-gym-b894cbe-20260914-202306.sqsh" \
MOUNTS="/lustre:/lustre,$(pwd -P)/env.yaml:/opt/Gym/env.yaml:x-create=file,$(pwd -P)/results:/opt/Gym/results,$(pwd -P)/benchmarks/nemotron_3.5_super/vllm_configs/nemotron_3_ultra.sh:/opt/Gym/benchmarks/nemotron_3.5_super/vllm_configs/nemotron_3_ultra.sh:ro" \
bash benchmarks/nemotron_3.5_super/sbatch_external_vllm.sh \
    --config benchmarks/nemotron_3.5_super/eval_container_config.yaml \
    '++num_samples_in_parallel=24' \
    '++resume_from_cache=true' \
    '++critpt_benchmark_agent.responses_api_agents.critpt_agent.datasets=[]' \
    '++hle_benchmark_equivalence_llm_judge_simple_agent.responses_api_agents.simple_agent.datasets=[]'

The two dataset-list overrides exclude CRITPT and HLE. Remove the corresponding final argument to include either benchmark in a fresh experiment. EXPORT_TO_CSV=0 skips the optional exporter, whose Omniscience metric-key mismatch remains unresolved; evaluation JSON and final aggregate metrics are still written.

Run command — SWE-bench Verified, post-merge 0.29 (30 minutes)

Uses the same Ultra recipe, container, concurrency 24, and coupled P4/D4 allocation as Super, with the OpenCode agent and OpenSandbox configs. Run from the repository root with the existing OpenSandbox connection configured in env.yaml or the environment. The distinct experiment name keeps SWE-bench results separate; no CRITPT/HLE exclusions are needed because the Super config is not loaded.

EXPERIMENT_NAME="nemotron-3-ultra-mtp3-c24-v0290-postmerge-swebench-verified-$(date -u +%Y%m%d-%H%M%S)"

MODEL=/lustre/fsw/portfolios/llmservice/users/igitman/hf_models/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 \
VLLM_CONFIG=benchmarks/nemotron_3.5_super/vllm_configs/nemotron_3_ultra.sh \
EXPERIMENT_NAME="$EXPERIMENT_NAME" \
ROLLOUTS_FPATH="results/$EXPERIMENT_NAME/resumable.jsonl" \
EXPORT_TO_CSV=0 \
NUM_PREFILL_NODES=4 \
NUM_DECODE_NODES=4 \
VLLM_PD_DEPLOYMENT_MODE=coupled \
VLLM_SLURM_SEGMENT=4 \
SBATCH_ACCOUNT=nemotron_rl_algo \
SBATCH_PARTITION=batch_long \
SBATCH_QOS=normal \
SBATCH_TIME=2-00:00:00 \
SBATCH_GRES=gpu:4 \
SBATCH_EXPORT=ALL \
CONTAINER="/lustre/fsw/portfolios/llmservice/users/fsiino/repos/Gym-inkling-tuning/results/vllm/vllm-openai:v0.29.0___super-vl-evals-2a3c65b-ultra-with-gym-b894cbe-20260914-202306.sqsh" \
MOUNTS="/lustre:/lustre,$(pwd -P)/env.yaml:/opt/Gym/env.yaml:x-create=file,$(pwd -P)/results:/opt/Gym/results,$(pwd -P)/benchmarks/nemotron_3.5_super/vllm_configs/nemotron_3_ultra.sh:/opt/Gym/benchmarks/nemotron_3.5_super/vllm_configs/nemotron_3_ultra.sh:ro" \
bash benchmarks/nemotron_3.5_super/sbatch_external_vllm.sh \
    --config responses_api_models/vllm_model/configs/vllm_model.yaml \
    --config benchmarks/swebench/verified/opencode.yaml \
    --config nemo_gym/sandbox/providers/opensandbox/configs/opensandbox.yaml \
    '++num_samples_in_parallel=24' \
    '++resume_from_cache=true'

Each command reserves eight nodes / 32 GPUs; running both simultaneously requires 16 nodes / 64 GPUs. These are separate 30-minute validation jobs, not full-suite score runs. Check model/router readiness, live generation, tokens/sec/request, waiting, KV usage, and saved-output health; additionally check sandbox startup and agent activity for SWE-bench. Startup and sandbox preparation consume part of the allocation, so SWE-bench may save few completed attempts within this window. The launcher submits a separate job-scoped cleanup job for each evaluation.

Command controls and resume

Control	Valid values	Why this command uses it
`VLLM_PD_DEPLOYMENT_MODE`	`independent` (default), `coupled`	Branch-added opt-in for multi-node tiers; Ultra requires `coupled` because a complete replica does not fit on one four-GPU node.
`VLLM_SLURM_SEGMENT`	Unset or a positive integer	Branch-added control; overrides main's `SEGMENT`. If both are unset/empty, use all allocated nodes as one segment. `4` aligns each segment with a four-node tier, the placement used in successful Ultra runs.
`SEGMENT`	Unset or a positive integer	Main's equivalent segmentation control, honored when `VLLM_SLURM_SEGMENT` is unset/empty. No need to set both.
`SBATCH_TIME`	Slurm duration within account/partition limits	Branch makes the prior `04:00:00` default configurable. `00:30:00` bounds this tuning test; it is not enough for the full suite.
`VLLM_USE_V2_MODEL_RUNNER`	Launcher now explicitly sets `0`	Main selects V1, which was faster in our tests. This no longer needs a command argument; submitting `1` will be overridden by the launcher. V1 is version-specific and deprecated, so future containers need revalidation.
`SBATCH_EXPORT`	Slurm export setting; `ALL` here	Existing Slurm control; carries the run environment, including the fixed output path, into the allocation.
`++num_samples_in_parallel`	Positive integer	Existing Gym concurrency control. `24` improved aggregate generation relative to `20` while averaging above 90 tokens/sec/request; it does not prove proportionally faster suite completion.
`ROLLOUTS_FPATH` + `++resume_from_cache`	Fixed output JSONL path; `true` or `false`	Existing controls. Set a unique mounted results path and `true` on the first submission to preserve completed work across Slurm requeues.

For a manual resume, wait until the previous job stops, replace the timestamp-generating EXPERIMENT_NAME assignment with its exact previous experiment name, and rerun with the same fixed output path, model, recipe, container, datasets, and resume override. Preserve both resumable.jsonl and resumable_materialized_inputs.jsonl; the latter records the selected tasks. A Slurm requeue reuses the submitted environment without needing that manual edit. Interrupted, unsaved work may be repeated; neither control configures Slurm's requeue policy.

Without ROLLOUTS_FPATH, the launcher chooses a new timestamped output path at each restart, so ++resume_from_cache=true alone will not find the prior results. Without resume enabled, Gym may clear the selected output. For any new tuning configuration or dataset selection, use a fresh experiment/output path rather than mixing results with a previous run.

Changes since review at `f656822f6`

Hardcoded the tuned settings: removed ULTRA_* switches and their validation branches. Prefill/decode memory utilization is 0.90/0.95, batched-token limits are 16384/8192, maximum sequences is 64, and MTP drafts three extra tokens. Graph modes, copied inputs, and decode capture sizes are fixed in the recipe.
Removed eager enforcement: both tiers use PIECEWISE graphs with copied inputs; no --enforce-eager flag or eager fallback remains.
Enabled prefix caching: removed the old disabling flag and warning. The original multiple-local-block assertion did not recur in the subsequent full vLLM 0.27.1 run or the short 0.29 tests. This is not a claim that all cache warnings or output-quality problems are fixed.
Applied async scheduling where supported by tuning: enabled on prefill, disabled on decode. Decode async was retested on 0.29 in 7153011; matching post-warmup durations gave 89.96 versus 95.55 tokens/sec/request with synchronous decode, approximately 6% slower. The reviewer recommendation was to use it if it helps, not require it on both tiers.
Reused existing concurrency/resume controls: removed NUM_SAMPLES_IN_PARALLEL and RESUME_EVAL_ON_REQUEUE. Commands now use Hydra's ++num_samples_in_parallel, a fixed ROLLOUTS_FPATH, and ++resume_from_cache=true. Launcher tests cover argument preservation and stable output paths across requeues and manual restarts.
Reconciled the latest main launcher changes: preserved main's V1 runner setting, router-default balancing thresholds, and cpu-normal cleanup QOS. Support both main's SEGMENT and our higher-priority VLLM_SLURM_SEGMENT, falling back to the calculated node count and emitting exactly one --segment argument. Coupled serving remains opt-in.

copy-pr-bot · 2026-08-21T03:26:35Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Allow experiments to override the speculative-token count while preserving the current default. Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Remove obsolete experiment switches and retain the validated MTP3 defaults. Simplify Slurm argument handling and correct the coupled decode health check. Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Detect prefill failures while waiting for decode readiness. Honor explicit concurrency and resume overrides over environment defaults. Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

fsiino-nvidia · 2026-09-09T22:12:58Z

/ok to test a26a1e2

Add connection and request timeouts so stalled probes cannot block prefill failure detection. Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Monitor prefill and router processes, propagate unexpected exits, and stop the surviving process. Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Cover override precedence, resume paths, health-check timeouts, and coupled-mode failure detection and cancellation cleanup. Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Test independent and coupled node roles, existing model settings, submission overrides, invalid inputs, and serving-only execution. Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Signed-off-by: Frankie Siino <fsiino@nvidia.com> # Conflicts: # benchmarks/nemotron_3.5_super/sbatch_external_vllm.sh

fsiino-nvidia · 2026-09-11T00:29:24Z

/ok to test f656822

bxyu-nvidia · 2026-09-11T03:48:16Z

+
+# The defaults reproduce the full-suite MTP3 configuration validated in Run
+# 060: eager prefill and piecewise decode CUDA graphs with graph-owned inputs.
+ULTRA_PREFILL_GPU_MEMORY_UTILIZATION="${ULTRA_PREFILL_GPU_MEMORY_UTILIZATION:-0.90}"


can we hardcode these once we have a roughly tuned config?

bxyu-nvidia · 2026-09-11T03:49:02Z

+        ;;
+esac
+
+case "$ULTRA_DECODE_CUDAGRAPH_COPY_INPUTS" in


same as above, can we hardcode this one things are tuned?

bxyu-nvidia · 2026-09-11T03:49:17Z

+    --distributed-timeout-seconds 3600
+    # NIXL-transferred Mamba state must not coexist with locally retained
+    # prefix-cache blocks; doing so triggers the multiple-local-block assertion.
+    --no-enable-prefix-caching


i think this should be fixed with more recent containers

Updated the container 0.27.1 -> 0.29 and reran.

bxyu-nvidia · 2026-09-11T03:49:28Z

+    --tensor-parallel-size 4
+    --no-async-scheduling
+    # Eager prefill avoids the compiled/CUDA-graph stalls observed during tuning.
+    --enforce-eager


we should not enforce eager because it's much slower

Removed --enforce-eager

bxyu-nvidia · 2026-09-11T03:49:38Z

+    --max-num-seqs "$ULTRA_MAX_NUM_SEQS"
+    --data-parallel-size-local 1
+    --tensor-parallel-size 4
+    --no-async-scheduling


we recently got async scheduling accuracy working!

Sounds good. I've enabled async scheduling on prefill, but kept it disabled on decode. Decode async was retested with vllm 0.29 but throughput was 89.96 versus 95.55 tokens/sec/request with synchronous decode.

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Enable prefix caching and piecewise graphs with async prefill and synchronous decode. Replace tuning overrides and eager fallback with settings tested in run 7091385. Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Signed-off-by: Frankie Siino <fsiino@nvidia.com> # Conflicts: # benchmarks/nemotron_3.5_super/sbatch_external_vllm.sh

fsiino-nvidia · 2026-09-15T04:58:32Z

/ok to test

copy-pr-bot · 2026-09-15T04:58:35Z

/ok to test

@fsiino-nvidia, there was an error processing your request: E1

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/1/

Signed-off-by: Frankie Siino <fsiino@nvidia.com> # Conflicts: # benchmarks/nemotron_3.5_super/sbatch_external_vllm.sh

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

fsiino-nvidia · 2026-09-19T03:26:31Z

/ok to test

copy-pr-bot · 2026-09-19T03:26:33Z

/ok to test

@fsiino-nvidia, there was an error processing your request: E1

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/1/

fsiino-nvidia · 2026-09-19T03:27:38Z

/ok to test ac30c28

github-actions · 2026-09-19T04:00:41Z

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/35420126531

github-actions · 2026-09-19T04:07:51Z

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/35420453311

fsiino-nvidia requested a review from bxyu-nvidia August 21, 2026 04:04

fsiino-nvidia added 5 commits September 8, 2026 20:01

Init ultra config

af94b07

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Tune Nemotron 3 Ultra vLLM config

62d9f78

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Add configurable MTP depth for Nemotron 3 Ultra tuning

cd9eb6a

Allow experiments to override the speculative-token count while preserving the current default. Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Add Ultra async scheduling and EPLB tuning controls

a75b4d1

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Support Ultra coupled prefill/decode serving

b894cbe

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

fsiino-nvidia force-pushed the fsiino/ultra-vllm-tuning branch from 36ba596 to b894cbe Compare September 8, 2026 20:27

fsiino-nvidia added 3 commits September 9, 2026 18:52

Clean up Ultra vLLM tuning config

f6387b1

Remove obsolete experiment switches and retain the validated MTP3 defaults. Simplify Slurm argument handling and correct the coupled decode health check. Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Fix vLLM startup monitoring and override precedence

e1628e9

Detect prefill failures while waiting for decode readiness. Honor explicit concurrency and resume overrides over environment defaults. Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Restore comment from main

a6c2a44

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

fsiino-nvidia marked this pull request as ready for review September 9, 2026 21:32

Merge remote-tracking branch 'origin/main' into fsiino/ultra-vllm-tuning

a26a1e2

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

fsiino-nvidia changed the title ~~Fsiino/ultra vllm tuning 08202026~~ Ultra 3 vllm tuning Sep 9, 2026

copy-pr-bot Bot temporarily deployed to public September 9, 2026 22:13 Inactive

copy-pr-bot Bot temporarily deployed to public September 9, 2026 22:15 Inactive

github-actions Bot added the sla:review-overdue Review response is over the one-business-day SLA label Sep 9, 2026

github-actions Bot mentioned this pull request Sep 9, 2026

[Tracker] Pull request handoff SLA #3158

Open

fsiino-nvidia added 5 commits September 10, 2026 22:09

Bound coupled vLLM startup health checks

b1da0ca

Add connection and request timeouts so stalled probes cannot block prefill failure detection. Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Detect coupled vLLM failures after startup

0ad4478

Monitor prefill and router processes, propagate unexpected exits, and stop the surviving process. Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Add regression tests for the Super vLLM launcher

55933f1

Cover override precedence, resume paths, health-check timeouts, and coupled-mode failure detection and cancellation cleanup. Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Expand Super vLLM launcher regression coverage

195962d

Test independent and coupled node roles, existing model settings, submission overrides, invalid inputs, and serving-only execution. Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Merge remote-tracking branch 'origin/main' into fsiino/ultra-vllm-tuning

f656822

Signed-off-by: Frankie Siino <fsiino@nvidia.com> # Conflicts: # benchmarks/nemotron_3.5_super/sbatch_external_vllm.sh

copy-pr-bot Bot temporarily deployed to public September 11, 2026 00:29 Inactive

copy-pr-bot Bot temporarily deployed to public September 11, 2026 00:31 Inactive

copy-pr-bot Bot temporarily deployed to public September 11, 2026 00:32 Inactive

bxyu-nvidia requested changes Sep 11, 2026

View reviewed changes

github-actions Bot removed the sla:review-overdue Review response is over the one-business-day SLA label Sep 11, 2026

fsiino-nvidia added 2 commits September 11, 2026 16:57

Prefer Hydra overrides for resuming and concurrency

21a0a10

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Pin Ultra's non-eager MTP3 tuning baseline

b4d8cf5

Enable prefix caching and piecewise graphs with async prefill and synchronous decode. Replace tuning overrides and eager fallback with settings tested in run 7091385. Signed-off-by: Frankie Siino <fsiino@nvidia.com>

github-actions Bot added the sla:author-overdue Author response is over the one-business-day SLA label Sep 14, 2026

Merge remote-tracking branch 'origin/main' into fsiino/ultra-vllm-tuning

173c5f3

Signed-off-by: Frankie Siino <fsiino@nvidia.com> # Conflicts: # benchmarks/nemotron_3.5_super/sbatch_external_vllm.sh

github-actions Bot removed the sla:author-overdue Author response is over the one-business-day SLA label Sep 15, 2026

fsiino-nvidia added 3 commits September 18, 2026 19:34

Merge remote-tracking branch 'origin/main' into fsiino/ultra-vllm-tuning

7551c06

Signed-off-by: Frankie Siino <fsiino@nvidia.com> # Conflicts: # benchmarks/nemotron_3.5_super/sbatch_external_vllm.sh

revert

11f2d90

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Merge remote-tracking branch 'origin/main' into fsiino/ultra-vllm-tuning

ac30c28

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

copy-pr-bot Bot temporarily deployed to public September 19, 2026 03:28 Inactive

nemo-automation-bot Bot requested review from a team September 19, 2026 03:28

copy-pr-bot Bot temporarily deployed to public September 19, 2026 03:29 Inactive

copy-pr-bot Bot temporarily deployed to public September 19, 2026 03:30 Inactive

bxyu-nvidia approved these changes Sep 19, 2026

View reviewed changes

bxyu-nvidia added this pull request to the merge queue Sep 19, 2026

Merged via the queue into main with commit dba68bb Sep 19, 2026
42 checks passed

bxyu-nvidia deleted the fsiino/ultra-vllm-tuning branch September 19, 2026 04:20

natsala13 mentioned this pull request Oct 5, 2026

feat(eval): add bounded cached continuation to direct jobs #4037

Draft

Conversation

fsiino-nvidia commented Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does

Benchmark scores

Run command — Super suite, post-merge 0.29 (30 minutes)

Run command — SWE-bench Verified, post-merge 0.29 (30 minutes)

Command controls and resume

Changes since review at f656822f6

Uh oh!

copy-pr-bot Bot commented Aug 21, 2026

Uh oh!

fsiino-nvidia commented Sep 9, 2026

Uh oh!

fsiino-nvidia commented Sep 11, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fsiino-nvidia commented Sep 15, 2026

Uh oh!

copy-pr-bot Bot commented Sep 15, 2026

Uh oh!

fsiino-nvidia commented Sep 19, 2026

Uh oh!

copy-pr-bot Bot commented Sep 19, 2026

Uh oh!

fsiino-nvidia commented Sep 19, 2026

Uh oh!

github-actions Bot commented Sep 19, 2026

Uh oh!

github-actions Bot commented Sep 19, 2026

Uh oh!

Uh oh!

This branch was previously deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

fsiino-nvidia commented Aug 21, 2026 •

edited

Loading

Changes since review at `f656822f6`