Repository navigation
Conversation
…Cm 7.14 Dockerfile (vllm-project#418) * [BugFix][ROCm] Widen rollout engine's visible devices so trainer IPC handles resolve on AMD GPUs A HIP IPC handle carries the EXPORTING process's device ordinal, and hipIpcOpenMemHandle resolves that ordinal against the IMPORTING process's device list. CUDA instead honours the importer's current device. Colocate weight sync therefore breaks on ROCm whenever a rollout engine is launched with a mask narrower than the trainer's: the trainer sees the whole node and exports at ordinal N, while the engine enumerates a single device where ordinal N is out of range. Weight sync succeeds on GPU 0 and fails on every other GPU with hipErrorInvalidValue. _compute_server_args built the engine subprocess's _visible_devices from the engine's own GPUs alone. Route it through _engine_visible_devices(), which lists the engine's own GPUs first and the rest of the node's afterwards. The engine still lands on its intended devices because its own are first; only the mask's LENGTH matters for ordinal resolution, not any agreement between the two numberings. Gated on torch.version.hip: on CUDA the narrow mask is correct and widening it would be a needless behaviour change. Verified on 8x MI308X (gfx942), ROCm 7.14, torch 2.12.0+rocm7.14, vLLM v0.29.0, colocate vLLM rollout + Megatron trainer. Single-variable A/B with 4 actor GPUs and --rollout-num-gpus-per-engine 1, so the engines are launched narrower than the trainer: without this commit three of four engines fail the first weight sync with a 500 from /update_weights and CUDA error: invalid argument in rebuild_cuda_tensor -> UntypedStorage._new_shared_cuda; with it the run exits 0 and all four engines sync. A framework-free two-process probe isolates the same ordinal resolution in ~10s. Co-authored-by: Claude Co-authored-by: Cursor <cursoragent@cursor.com> Signed-off-by: Treemann <rongzhang.zheng@amd.com> * [Docker][ROCm] Add a ROCm 7.14 Dockerfile for AMD GPUs docker/Dockerfile.rocm targets ROCm 7.0.2 / gfx950 and builds the whole stack onto ubuntu:22.04 across nine stages. On a host that already has a prebuilt ROCm 7.14 image most of that work is done, so this variant starts from rocm/primus:v26.4 - which ships torch, triton, flash-attention, TransformerEngine and aiter - and adds only vLLM, Megatron-LM and vime. Single stage, no build context: every source is a pinned upstream git coordinate with its commit asserted, and nothing is COPYed in. VIME_REF accepts a branch or a full 40-hex sha; a sha is asserted, a branch has its resolved sha printed. Two properties are deliberate. The base image's ROCm torch is never replaced. A pip constraints file is generated from the installed versions right after the vLLM build, and every later pip install runs under it. vime's requirements.txt asks for vllm-router, transformers and ray[default] unpinned; without the constraints a resolver can satisfy one of them with a PyPI torch or vllm wheel - those are CUDA-only - and the breakage surfaces much later as a runtime failure on the first GPU op. A pip install --dry-run guard aborts the build if the resolver still plans to do it. A verification layer fails the build rather than the first training run: ROCm torch untouched, VLLM_TAG actually installed, vime/Megatron/Ray/ torch_memory_saver importing, and the vLLM API window vime needs present at both ends. That last check is why the VLLM_TAG pin is a checked claim and not a comment: vime needs NCCLTrainerInitInfo from vllm.distributed.weight_transfer.nccl_engine, which v0.27.1 predates, and it needs whichever cli_args location its current revision imports - the check reads that from vime's own source so it stays meaningful as main advances. Additive: docker/Dockerfile.rocm is untouched. Co-authored-by: Claude Co-authored-by: Cursor <cursoragent@cursor.com> Signed-off-by: Treemann <rongzhang.zheng@amd.com> * Remove the unnecessary max() operation Signed-off-by: Rongzhang Zheng <rongzhang.zheng@amd.com> * [Docker][ROCm] Fix --offload-train issues in rocm7.14 image - Bump torch_memory_saver to 016938275e46e72e7b8d60e8d15046171b3e3c72 to align HIP VMM allocation sizes to allocation granularity - Set ROC_ENABLE_LARGE_BAR=0 to avoid TMS VMM .item() SIGSEGV on MI300-class hosts - Canonicalize duplicate libamd_comgr.so.3 under _rocm_sdk_* and fail the build if LD_PRELOADing the TMS hook still triggers duplicate LLVM registration on import torch Signed-off-by: Treemann <rongzhang.zheng@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com> --------- Signed-off-by: Treemann <rongzhang.zheng@amd.com> Signed-off-by: Rongzhang Zheng <rongzhang.zheng@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
…roject#441) * sync: update from Slime through #2393 Signed-off-by: aoshen02 <aoshen@inferact.ai> * fix: complete vLLM sync translations Signed-off-by: aoshen02 <aoshen@inferact.ai> * sync: complete Slime update through #2394 Signed-off-by: aoshen02 <aoshen@inferact.ai> * Sync Slime through score centering Signed-off-by: aoshen02 <aoshen@inferact.ai> * fix: rebase vLLM patches onto current nightly Signed-off-by: aoshen02 <aoshen@inferact.ai> * fix: remove stale async CI entry and obsolete metrics fallbacks Signed-off-by: aoshen02 <aoshen@inferact.ai> * feat: transport sampling-mask logprobs for score centering Signed-off-by: aoshen02 <aoshen@inferact.ai> * fix(vllm): enable scale-out endpoint for rollout TITO Signed-off-by: aoshen02 <aoshen@inferact.ai> * fix(rollout): keep requests during sleep and enable external PD TITO Signed-off-by: aoshen02 <aoshen@inferact.ai> * fix(ci): enable TITO on external OPD teacher Signed-off-by: aoshen02 <aoshen@inferact.ai> * test: enable scale-out render endpoint in Geo3K e2e Signed-off-by: aoshen02 <aoshen@inferact.ai> * feat: mirror Slime persistent rollout queue and distributed fully async Signed-off-by: aoshen02 <aoshen@inferact.ai> * test: exercise straw transport in mirrored GPU suites Signed-off-by: aoshen02 <aoshen@inferact.ai> * style: match repository formatter for straw sync Signed-off-by: aoshen02 <aoshen@inferact.ai> * ci: install straw in CPU contract container Signed-off-by: aoshen02 <aoshen@inferact.ai> * sync: mirror Slime #2427 straw checkpoint rollback Signed-off-by: aoshen02 <aoshen@inferact.ai> * ci: install straw dependency for sync tests Signed-off-by: aoshen02 <aoshen@inferact.ai> * test: pin vllm generation config for score centering Signed-off-by: aoshen02 <aoshen@inferact.ai> * fix: bind straw coordinator in vllm rollout Signed-off-by: aoshen02 <aoshen@inferact.ai> * docs: record full GPU gate procedure Signed-off-by: aoshen02 <aoshen@inferact.ai> * sync: mirror Slime #2432 Straw queue and replay changes Signed-off-by: aoshen02 <aoshen@inferact.ai> * ci: quote Straw requirement in Buildkite shell steps Signed-off-by: aoshen02 <aoshen@inferact.ai> * [Sync] Mirror Slime PipelineRL and reference top-p fix Port #2437 flush scheduling and #2434 reference log-prob replay fix. Replace request-bound TITO weight version with per-output propagation through vLLM #53199, and preserve exact top-p score-centering support. Signed-off-by: aoshen02 <aoshen@inferact.ai> * [CI] Add flush interval to weight updater test fixtures Signed-off-by: aoshen02 <aoshen@inferact.ai> * [CI] Accept versionless terminal abort chunk in PipelineRL probe Signed-off-by: aoshen02 <aoshen@inferact.ai> * [Sync] Mirror concurrent rollout and HTTP regression tests Signed-off-by: aoshen02 <aoshen@inferact.ai> * [Sync] Preserve buffered-token weight version on abort Signed-off-by: aoshen02 <aoshen@inferact.ai> * [Sync] Avoid zero-KL assertion for top-p actor/ref mismatch Signed-off-by: aoshen02 <aoshen@inferact.ai> * [Sync] Backport AuxOutput reset fix from vLLM #59060 Signed-off-by: aoshen02 <aoshen@inferact.ai> * [Sync] Stamp synthetic abort outputs with committed version Signed-off-by: aoshen02 <aoshen@inferact.ai> * [Sync] Preserve coalesced rollout metadata and validate sampler probabilities Signed-off-by: aoshen02 <aoshen@inferact.ai> * [Sync] Remove superseded per-file spill and mirror Straw references Signed-off-by: aoshen02 <aoshen@inferact.ai> * [CI] Inherit the available file descriptor limit for GPU jobs Signed-off-by: aoshen02 <aoshen@inferact.ai> * [Sync] Backport terminal token-stream finish reasons Signed-off-by: aoshen02 <aoshen@inferact.ai> * [Test] Cover terminal output with real prompt token metadata Signed-off-by: aoshen02 <aoshen@inferact.ai> * [Sync] Mirror latest Slime and validate nightly Python/Rust adapters Mirror Slime through 2f231865 with approved native overlays and Source test registration. Pin validated43b nightly, install tested tensor-dump/tracing Python and Rust patches, and parse native integer-ID top-k responses. Preserve original Straw assertions and saving policy. Keep profiler deferred and latest unchanged. AI assistance: OpenAI Codex. Scoped validation171CPU tests, full precommit and11patch/98path installation proof pass; full remote CI and remaining candidate GPU validation pending. Signed-off-by: aoshen02 <aoshen@inferact.ai> * [Sync] Mirror Slime external PD delta artifact assertion Signed-off-by: aoshen02 <aoshen@inferact.ai> * fix: repair verified recovery and initial-policy CI contracts Preserve training loss, numerical thresholds and checkpoint strategy. Add indexed precision dumps, offline-first manager setup, scoped reference load, same-distribution CI KL diagnostic, canonical optimizer masters and NIXL peer retirement. Signed-off-by: aoshen02 <aoshen@inferact.ai> * fix: keep initial-model diagnostic optional for direct loss callers Signed-off-by: aoshen02 <aoshen@inferact.ai> * fix: give the PD rollout debug dump a unique per-rollout path Signed-off-by: aoshen02 <aoshen@inferact.ai> * [Sync] Mirror Slime v0.4.0 training stack and release updates Signed-off-by: aoshen02 <aoshen@inferact.ai> * [Sync] Mirror latest Slime documentation and native engine contracts Signed-off-by: aoshen02 <aoshen@inferact.ai> * [Sync] Restore upstream documentation release checks Signed-off-by: aoshen02 <aoshen@inferact.ai> * fix(sync): mirror active DeepEP recipes and docs CI contracts Signed-off-by: aoshen02 <aoshen@inferact.ai> * docs(sync): mirror Slime environment entry and preserve sampler parameters Signed-off-by: aoshen02 <aoshen@inferact.ai> * [Sync] Preserve upstream export explanation and tokenization attribution Signed-off-by: aoshen02 <aoshen@inferact.ai> * [Sync] Inherit Slime documentation deletions and navigation Signed-off-by: aoshen02 <aoshen@inferact.ai> * Remove opt-in DeepEP auto switching Signed-off-by: aoshen02 <aoshen@inferact.ai> * [RL] Release native weight-transfer communicators and expose lifecycle APIs Include fork vLLM vllm-project#119 on the fixed 43b4 baseline, including explicit communicator destruction and matching Python/Rust public controls. Keep the other sixteen-patch capabilities intact. Assisted-by: OpenAI Codex Signed-off-by: aoshen02 <aoshen@inferact.ai> * [Sync] Mirror Slime agent and collective updates through 58b08786 Source PRs: THUDM/slime#2453, #2450, #2458, #2457. Preserve approved Vime overlays and restore the omitted Source Straw error assertions. Native TITO, replay/SC, MTP and Buildkite adapters use the fixed 43b4 engine stack. Validation: Slurm2366 301CPU/1skip,2376 Python3.11 agent step163pass/2optional skips,2377 Straw7+88pass,2378 nine precommit hooks. Native sandbox SSD prerequisites2373 pass; full new8GPU/model and final-image gates remain pending. No latest or release promotion. Slime-commit: 58b0878679de339ec7844b45d884a87955a134be Co-authored-by: Codex <noreply@openai.com> Signed-off-by: aoshen02 <aoshen@inferact.ai> * [Build] Package native ARM Sunabako without changing x86 installation Preserve Slime SDK0.1.1 contracts and the original x86 build target. Build a verified native ARM wheel with matching seccomp/syscall/ABI and driver paths before requirements resolution. Source-Slime: 58b0878679de339ec7844b45d884a87955a134be Co-authored-by: Codex <noreply@openai.com> Signed-off-by: aoshen02 <aoshen@inferact.ai> * Mirror rollout weight-group lifecycle and preserve draft state across sleep Restore Source connect/disconnect layout and Agent MTP translation. Reuse vLLM parameter retention for draft-private weights instead of adding a second backup switch. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: aoshen02 <aoshen@inferact.ai> * sync: align draft retention patch with reviewed native selector Mirror the simplified native proposal, including an empty fully-shared draft. No trainer, saving, or inference-path policy change. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: aoshen02 <aoshen@inferact.ai> * [Sync] Mirror router health-check ownership with native gateway patch Signed-off-by: aoshen02 <aoshen@inferact.ai> * fix(sync): mirror source documentation and MTP verification budgets Co-authored-by: Codex <noreply@openai.com> Signed-off-by: aoshen02 <aoshen@inferact.ai> * fix(rust): mirror multimodal render-to-generate support Keep the pinned vLLM base and existing Rust overlays; apply the independent multimodal bridge after them. Preserve explicit UUIDs, native validation and metadata-only EC support. Remove the GB10 launcher resurrected after the approved platform deletion. Signed-off-by: aoshen02 <aoshen@inferact.ai> * fix(epd): forward producer transfer metadata and preserve media identity Use the existing rollout endpoint metadata instead of inert SGLang-only engine keys. Preserve ordinary/streaming placement, complete media parts and caller UUIDs. Do not alias language_only to the unrelated modality-limit flag. Validation: frozen700-path CPU2502 passes92 cases; real two-host ECCPU/NIXL2503 passes both frontends with four actual transfers and baseline-exact tokens/UUID metadata-only. Explicit connector/body assembly, not default deployment or all-model certification. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: aoshen02 <aoshen@inferact.ai> * fix(epd): mirror encoder deployment with native EC transfer Keep Slime deployment sequencing and consumer rendering. Use native EC transport and DP endpoint ownership, preserve decoder input processing while omitting encoder weights, and supply resolved engine config and embedding support to Rust. Reuse existing upstream authentication fix; retain test assertions. Validation: complete 23-patch stack on43b;83CLI/468server and all510Rust paths; installed experimentalfefcf78runtime/705sourcepaths/92CPU; no-overlay original3GPU E-P-D2552 passes Python/Rust. One-hour2554 is running, not claimed complete. Native review worktrees remain uncommitted. AI assistance used. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: aoshen02 <aoshen@inferact.ai> --------- Signed-off-by: aoshen02 <aoshen@inferact.ai> Co-authored-by: Codex <noreply@openai.com>
…ants (vllm-project#474) * ci(rocm): ROCm CI test suite + review fixes, rebased onto main Rebased onto upstream main (which now carries the Slime v0.4.0 sync vllm-project#441: rebased vLLM patches, enable_scale_out, test_accelerator changes). Drops the now-redundant docker/patch/vllm_engine content this branch previously copied from vllm-project#432 — main and vllm-project#432 own those. Keeps only the ROCm CI additions and the PR #3 review fixes: - .buildkite/pipeline-rocm.yaml: RAY_EXPERIMENTAL_NOSET_* + NCCL env, the HIP->CUDA visibility wrapper (normalizes CUDA_VISIBLE_DEVICES to the locked HIP set in-command, keeping tests/ci/gpu_lock_exec.py untouched), and the vllm-config/quick-start test steps. - tests/*-rocm.py: 8 standalone ROCm test variants (self-contained U.is_rocm shim; experimental/unwired PPO-critic note; corrected mixed-offload level-1-sleep restore description; dead code removed). - docs amd_tutorial: `sleep infinity` so detached container stays up for exec. Co-Authored-By: Claude <noreply@anthropic.com> * ci(rocm): keep green vllm-config suite; bundle validated ROCm image Retest on the rebuilt image (vllm/vime-rocm:pr-441) cleared startup and the health gate for all suites; the 4 vllm-config GRPO tests pass to a train step. Drop the 4 that fail for reasons outside this CI work: - quick_start_glm4_9B: GLM-specific step-0 KL exceeds the strict <1e-8 assert (ROCm rollout/train logprob divergence; the 0.5B GRPO tests pass the same check) - qwen3_4B_ppo{,_train_critic_only,_disaggregate}: SIGSEGV in critic model creation (known experimental ROCm PPO-critic path) Bundle the validated Dockerfile.rocm (layers on vllm/vllm-openai-rocm:nightly, credit @indianspeedster vllm-project#432) with the fixes the retest surfaced: - pin pyarrow==25.0.1: the base's 26.0.0 is a numpy-2 build that rejects the numpy 1.26.4 pin at import, breaking Ray serialization + HF datasets - apply vllm-generation-health.patch: provides /health_generate, which the post-vllm-project#441 rollout health monitor polls (slim 3-patch set had dropped it) - drop unused megatron-energon / multi-storage-client - CMD sleep infinity so `docker run -d` stays up for exec Co-Authored-By: Claude <noreply@anthropic.com> * docker(rocm): isolate ROCm-rebased vLLM patches from the shared CUDA set d21bad2 overwrote docker/patch/latest/{vllm,vllm-pull_weights, vllm-inflight-queue-diagnostics}.patch with ROCm-rebased versions, but the CUDA docker/Dockerfile applies those same files with strict `git apply` — the rebase would break the CUDA image build. The ROCm nightly base genuinely needs the rebased patches: main's CUDA versions apply without rejects but silently misplace the weight_version hunk under --fuzz (lands on the wrong protocol class), failing the import-check assert. Resolve by isolation: - docker/patch/latest/ reverted to main's versions (CUDA untouched) - ROCm-rebased patches moved to docker/amd_patch/latest/ (ROCm-only dir, already home to the ROCm megatron patches); Dockerfile.rocm points there - vllm-generation-health.patch stays shared (applies clean on both bases) Rebuilt vllm/vime-rocm:pr-441 from docker/amd_patch/ — import check passes (weight_version correctly placed). Also trims Dockerfile/pipeline comment blocks to <=2 lines. Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
There was a problem hiding this comment.
Code Review
This pull request updates the Vime repository by introducing an interactive experiment builder lab to the documentation, restructuring the docs directory, and updating the Docker and Buildkite CI configurations to support new features like speculative decoding, fault tolerance, and straw transport. Feedback on the changes includes installing the missing patch utility in docker/Dockerfile.rocm to prevent build failures and updating a legacy [slime-docs] prefix to [vime-docs] in docs/build.sh for consistency.
| RUN apt-get update && apt-get install -y \ | ||
| rsync dnsutils iproute2 prometheus && \ | ||
| rm -rf /var/lib/apt/lists/* |
There was a problem hiding this comment.
The patch utility is used later in the Dockerfile (lines 126-129) to apply various vLLM patches, but it is not installed in this stage. Since the base image vllm/vllm-openai-rocm:nightly may not have patch pre-installed, please add patch to the apt-get install command to prevent potential build failures.
RUN apt-get update && apt-get install -y \
rsync dnsutils iproute2 prometheus patch && \
rm -rf /var/lib/apt/lists/*
| if [ -z "$SPHINX_BUILD" ]; then | ||
| cat >&2 <<'EOF' | ||
| [slime-docs] sphinx-build was not found. From the repository root, run: | ||
| python3 -m venv .venv-docs | ||
| .venv-docs/bin/python -m pip install -r docs/requirements.txt | ||
| Then retry: bash docs/build_all.sh | ||
| EOF | ||
| exit 1 | ||
| fi |
There was a problem hiding this comment.
The error message refers to [slime-docs], which is the old name of the project. Since the project has been renamed to Vime, please update this prefix to [vime-docs] to maintain consistency with other build scripts (e.g., docs/build_all.sh uses [vime-docs]).
| if [ -z "$SPHINX_BUILD" ]; then | |
| cat >&2 <<'EOF' | |
| [slime-docs] sphinx-build was not found. From the repository root, run: | |
| python3 -m venv .venv-docs | |
| .venv-docs/bin/python -m pip install -r docs/requirements.txt | |
| Then retry: bash docs/build_all.sh | |
| EOF | |
| exit 1 | |
| fi | |
| if [ -z "$SPHINX_BUILD" ]; then | |
| cat >&2 <<'EOF' | |
| [vime-docs] sphinx-build was not found. From the repository root, run: | |
| python3 -m venv .venv-docs | |
| .venv-docs/bin/python -m pip install -r docs/requirements.txt | |
| Then retry: bash docs/build_all.sh | |
| EOF | |
| exit 1 | |
| fi |
Summary
Brings
0z5a-branchup tomain(d7e40fe) with one merge commit (c1027a5f, signed off), so later work can be based onmainand reviewed incrementally against this branch.The merge adds the three
maincommits that0z5a-branchlacked:4b4acac(MCore 0.19) and its MTP module-name mapping;How conflicts were resolved
mainis the reference. Every conflicting hunk takesmain's version (git merge -X theirs origin/main). Changes that exist only on0z5a-branchand do not conflict are kept:docker/is identical tomain.Differences from
mainafter this merge45 files, +2942 / −48, all from #454, #459 and #460:
vime/utils/draft_feature_contract.py,vime/backends/megatron_utils/draft_feature_collector.py, actor/arguments hooks, docsadvanced/draft-feature-collection.md,tests/test_draft_feature_*.py,tests/integration/run_draft_feature_ouro.pyvime_plugins/ouro/,examples/ouro/, Ouro loader and exporter entries inhf_to_megatron/megatron_to_hfvime/rollout/vllm_rollout.py,vime/utils/arguments.py,vime/utils/reward_normalization.py,vime/utils/tensor_backper.py,vime/observability/*,examples/mem_agent/_common.sh, docsget_started/usage.md,tests/test_server_idle_contract.py,tests/test_mem_agent_nvlink_detect.py,tests/integration/test_native_adam_resume.pyCI, as differences from
mainI ran the 44 CPU test files listed in
.buildkite/pipeline.ymlonmainand on this branch:main.tests/test_megatron_argument_validation.py: onmain, 71 passed. Here, 87 passed and 22 failed. All 22 failing tests exist only on0z5a-branch:test_colocate_offload_resolution(18 cases, [Maintenance] Consolidate rollout, checkpoint, topology fixes and colocate docs #460);test_collect_only_supported_argumentsandtest_collect_only_rejects_before_side_effects(4 cases, [Feature][Training] Collect versioned target features for external draft #454).main's logic won the conflict invime/utils/arguments.py: colocate offload defaults, and validation of the collect-only mode.mainis follow-up work.d7e40fe8([CI][ROCm] Add ROCm CI: Dockerfile, Buildkite pipeline, and test variants #474) has noSigned-off-byonmainitself; this PR does not change it. The merge commit is signed off.GPU tests were not run for this merge.
AI assistance
The merge, the checks and this description were made with Claude Code (AI-assisted).
🤖 Generated with Claude Code