[0z5a]

Purpose

Validate rollout data before Sample mutation, restore Adam state with independent storage, and consolidate topology, request metrics, bounded drain and colocate fixes. Includes local Torch Qwen norm conversion and pageable CPU weight backups on WSL.

Target: 0z5a-branch at 3e569314. Maintenance increment: 25 files, +1,709/-521; head 9a7cb7ec.

Consolidates #426, #427, #438, #439, #440, #444 and #445; #443 remains independent. Collector #454 and TransferQueue #453 are downstream stacks. Existing PRs remain Draft. Private validation scripts, logs, model tensors and coordination records are outside the diff.

Test Plan

Official Qwen3-0.6B (c1899de2), BF16, RTX 5080 SM120, Torch 2.13+cu132. Three native GRPO updates versus two saved updates plus a fresh-process third update: complete typed model/optimizer/scheduler/RNG state, data cursor, tokens/rewards/gradients and physical serving weights. Each rollout has 16 samples and at most 16 response tokens. Repeat independently with original NVIDIA Apex FusedAdam.

Both comparison stacks share scalar Torch AdamW (foreach=False), FlashDotProductAttention, pageable DCP staging, Qwen norm/WSL compatibility fixes and native full/disk HF publication. The private external frontend bridge invokes original level-2 release and weights/KV resume RPCs, checking actual HTTP sleep/wake. Vendor validation uses original Apex optimizer/kernels with Torch RMSNorm.

Three complete AB/BA/AB pairs compare separately built base vLLM 9236159 plus its four original patches with candidate ced6857 plus current patches. Both use native V1 (VLLM_USE_V2_MODEL_RUNNER=0): baseline V2 requires UVA unavailable on WSL. Ordinary/vendor restoration used candidate V2. Measurements cover two complete compatible stacks. Entry time includes private hashing audits; native update/publication spans exclude added hashing. Build time, persistent service startup and imports before train(args) are excluded.

Test Result

Validation	Result	Scope
Changed-path CPU suite	179 passed, 0 skipped	Before the two model follow-ups; rollout/drain/topology/arguments/docs/HF mapping
Current HF conversion suite	24 passed, 0 skipped	Local norm round-trip; CUDA uninitialized
Malformed-rollout base/candidate	13 failed / 2 passed → 15 passed	Identical boundary cases; candidate included above
Native CUDA optimizer restore	10 passed	Native/fused Torch Adam/AdamW and actual CPU/GPU Hybrid; shared Thor
Pinned Apex common-state restore	PASS, 6 natural exits	Adam/AdamW continuous4 vs 2-save+fresh2; 688 fields and 9 typed arrays exact; optimizer fixture
Base/candidate pinned vLLM builds	PASS, both complete source builds exit0	Four patches each, CUDA/Rust builds and five owned core extensions each verified; 14 actual patched metrics/checkpoint tests pass
Packed attention / WSL transport	PASS	BF16 GQA forward/backward, native one-layer model, CPU/GPU weight backups and complete-size DCP staging
Full Qwen ordinary AdamW train/save/resume	PASS, all 6 driver/frontend exits0	Continuous3 vs split2+fresh1; all 1,048 typed DCP chunks per round, optimizer/scheduler/RNG/data and physical publications exact; 8,346,329,812 payload bytes per saved iteration
Full Qwen Apex FusedAdam train/save/resume	PASS, all 6 driver/frontend exits0	Same complete restore audit and full/disk transport; original vendor optimizer/kernels
Real NIXL / Qwen P/D / HTTP drain	PASS	Three real READ/WRITE and three NIXL+Offloading P/D rounds, exact KV/output and both active drain phases; shared one-GPU Thor/TCP loopback, earlier compiled P/D extensions
Earlier full Ouro BF16 train/resume	PASS	Three updates vs two-save+fresh1; 1,074 typed DCP chunks exact on A100 40GB; relevant production files unchanged
Docs and automated CI	PASS	EN/ZH, Buildkite #1423, DCO/docs at current head; no manual reruns
Matched complete Qwen runs	PASS, 3 pairs	All six driver/frontend pairs naturally exit0, all actors self-exit, three updates/saves per run, physical publications and per-sample prompt/response lengths match

Matched measurement (seconds)	Base median	Candidate median	Base/candidate ratio	Duration change	Base / candidate range
Complete native entry, including audits	647.103	643.414	1.006×	-0.57%	646.728–654.692 / 643.210–644.813
Three native training updates	56.374	56.314	1.001×	-0.11%	56.335–61.169 / 56.302–56.362
Three native full/disk publications	28.511	28.329	1.006×	-0.64%	28.402–29.319 / 27.770–29.032

Observed duration reductions are small (0.11–0.64%); these three pairs establish the measured scope, without a general speedup or VIME-only attribution.

Ordinary gradient norms: 17.4246958549 / 16.9210435940 / 17.6251870003; vendor: 17.4246958549 / 16.9137404343 / 17.6507095151. Across the six speed runs, maximum sample-mean logprob difference is 0.00643478 (threshold 0.03), maximum individual-token difference 0.21428418. Exact BF16 train/inference logprob equality is not established.

Transformer Engine is untested; plain Torch CUDA IPC is unavailable on this WSL host. The PR remains Draft.

[read-the-docs-community]

Documentation build overview

📚 vime | 🛠️ Build #34924109 | 📁 Comparing 9a7cb7e against latest (791f101)

  🔍 Preview build  

46 files changed · + 1 added · ± 45 modified

+ Added

• _examples_synced/ouro/README.html

± Modified

• genindex.html
• index.html
• search.html
• advanced/arch-support-beyond-megatron.html
• advanced/delta-weight-sync.html
• advanced/external-rollout-engines.html
• advanced/fault-tolerance.html
• advanced/low-precision.html
• advanced/megatron-config.html
• advanced/observability.html
• and 35 more...

[gemini-code-assist]

Code Review

This pull request introduces several enhancements and robustness fixes across the codebase. Key changes include patching Megatron's distributed optimizer to correctly handle native Adam/AdamW state restoration, implementing a robust rollout server-idle completion check with parallel engine probing, and refining NVLink detection to prevent launcher aborts under strict shell options. Additionally, it updates argument validation for colocated memory offloading, improves vLLM generation metadata validation, and adds comprehensive unit and integration tests. Feedback on the changes suggests adding a pytestmark to skip the new CUDA-dependent integration tests when running in CPU-only environments to prevent runtime failures.

[gemini-code-assist]

Since this test file utilizes CUDA devices (device="cuda" on line 22), running it in a CPU-only environment (such as local development machines or standard CPU CI runners) will raise a RuntimeError. To make the test suite more robust and prevent unexpected failures on non-GPU environments, it is highly recommended to add a module-level pytestmark to skip all tests in this file if CUDA is not available.

Suggested change

	NUM_GPUS = 1
	pytestmark = pytest.mark.skipif(not torch.cuda.is_available(), reason="CUDA is required for this test")
	NUM_GPUS = 1