Skip to content

[Sync] Update 0z5a-branch to main (d7e40fe8) - #483

Open
0z5a wants to merge 4 commits into
vllm-project:0z5a-branchfrom
0z5a:sync-0z5a-branch-to-main
Open

0z5a wants to merge 4 commits into
vllm-project:0z5a-branchfrom
0z5a:sync-0z5a-branch-to-main

Conversation

@0z5a

@0z5a 0z5a commented Oct 19, 2026

Copy link
Copy Markdown

Summary

Brings 0z5a-branch up to main (d7e40fe) with one merge commit (c1027a5f, signed off), so later work can be based on main and reviewed incrementally against this branch.

The merge adds the three main commits that 0z5a-branch lacked:

How conflicts were resolved

main is the reference. Every conflicting hunk takes main's version (git merge -X theirs origin/main). Changes that exist only on 0z5a-branch and do not conflict are kept:

docker/ is identical to main.

Differences from main after this merge

45 files, +2942 / −48, all from #454, #459 and #460:

Area Files
Draft-feature collection (#454) vime/utils/draft_feature_contract.py, vime/backends/megatron_utils/draft_feature_collector.py, actor/arguments hooks, docs advanced/draft-feature-collection.md, tests/test_draft_feature_*.py, tests/integration/run_draft_feature_ouro.py
Ouro (#459) vime_plugins/ouro/, examples/ouro/, Ouro loader and exporter entries in hf_to_megatron / megatron_to_hf
Consolidation (#460) vime/rollout/vllm_rollout.py, vime/utils/arguments.py, vime/utils/reward_normalization.py, vime/utils/tensor_backper.py, vime/observability/*, examples/mem_agent/_common.sh, docs get_started/usage.md, tests/test_server_idle_contract.py, tests/test_mem_agent_nvlink_detect.py, tests/integration/test_native_adam_resume.py

CI, as differences from main

I ran the 44 CPU test files listed in .buildkite/pipeline.yml on main and on this branch:

GPU tests were not run for this merge.

AI assistance

The merge, the checks and this description were made with Claude Code (AI-assisted).

🤖 Generated with Claude Code

Treemann and others added 4 commits October 10, 2026 19:55
…Cm 7.14 Dockerfile (vllm-project#418)

* [BugFix][ROCm] Widen rollout engine's visible devices so trainer IPC handles resolve on AMD GPUs

A HIP IPC handle carries the EXPORTING process's device ordinal, and
hipIpcOpenMemHandle resolves that ordinal against the IMPORTING process's
device list. CUDA instead honours the importer's current device. Colocate
weight sync therefore breaks on ROCm whenever a rollout engine is launched
with a mask narrower than the trainer's: the trainer sees the whole node and
exports at ordinal N, while the engine enumerates a single device where
ordinal N is out of range. Weight sync succeeds on GPU 0 and fails on every
other GPU with hipErrorInvalidValue.

_compute_server_args built the engine subprocess's _visible_devices from the
engine's own GPUs alone. Route it through _engine_visible_devices(), which
lists the engine's own GPUs first and the rest of the node's afterwards. The
engine still lands on its intended devices because its own are first; only
the mask's LENGTH matters for ordinal resolution, not any agreement between
the two numberings.

Gated on torch.version.hip: on CUDA the narrow mask is correct and widening
it would be a needless behaviour change.

Verified on 8x MI308X (gfx942), ROCm 7.14, torch 2.12.0+rocm7.14, vLLM
v0.29.0, colocate vLLM rollout + Megatron trainer. Single-variable A/B with
4 actor GPUs and --rollout-num-gpus-per-engine 1, so the engines are launched
narrower than the trainer: without this commit three of four engines fail the
first weight sync with a 500 from /update_weights and CUDA error: invalid
argument in rebuild_cuda_tensor -> UntypedStorage._new_shared_cuda; with it
the run exits 0 and all four engines sync. A framework-free two-process probe
isolates the same ordinal resolution in ~10s.

Co-authored-by: Claude
Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Treemann <rongzhang.zheng@amd.com>

* [Docker][ROCm] Add a ROCm 7.14 Dockerfile for AMD GPUs

docker/Dockerfile.rocm targets ROCm 7.0.2 / gfx950 and builds the whole stack
onto ubuntu:22.04 across nine stages. On a host that already has a prebuilt
ROCm 7.14 image most of that work is done, so this variant starts from
rocm/primus:v26.4 - which ships torch, triton, flash-attention,
TransformerEngine and aiter - and adds only vLLM, Megatron-LM and vime.

Single stage, no build context: every source is a pinned upstream git
coordinate with its commit asserted, and nothing is COPYed in. VIME_REF
accepts a branch or a full 40-hex sha; a sha is asserted, a branch has its
resolved sha printed.

Two properties are deliberate.

The base image's ROCm torch is never replaced. A pip constraints file is
generated from the installed versions right after the vLLM build, and every
later pip install runs under it. vime's requirements.txt asks for
vllm-router, transformers and ray[default] unpinned; without the constraints
a resolver can satisfy one of them with a PyPI torch or vllm wheel - those
are CUDA-only - and the breakage surfaces much later as a runtime failure on
the first GPU op. A pip install --dry-run guard aborts the build if the
resolver still plans to do it.

A verification layer fails the build rather than the first training run: ROCm
torch untouched, VLLM_TAG actually installed, vime/Megatron/Ray/
torch_memory_saver importing, and the vLLM API window vime needs present at
both ends. That last check is why the VLLM_TAG pin is a checked claim and not
a comment: vime needs NCCLTrainerInitInfo from
vllm.distributed.weight_transfer.nccl_engine, which v0.27.1 predates, and it
needs whichever cli_args location its current revision imports - the check
reads that from vime's own source so it stays meaningful as main advances.

Additive: docker/Dockerfile.rocm is untouched.

Co-authored-by: Claude
Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Treemann <rongzhang.zheng@amd.com>

* Remove the unnecessary max() operation

Signed-off-by: Rongzhang Zheng <rongzhang.zheng@amd.com>

* [Docker][ROCm] Fix --offload-train issues in rocm7.14 image

- Bump torch_memory_saver to 016938275e46e72e7b8d60e8d15046171b3e3c72 to align HIP VMM
  allocation sizes to allocation granularity
- Set ROC_ENABLE_LARGE_BAR=0 to avoid TMS VMM .item() SIGSEGV on MI300-class hosts
- Canonicalize duplicate libamd_comgr.so.3 under _rocm_sdk_* and fail the build if
  LD_PRELOADing the TMS hook still triggers duplicate LLVM registration on import torch

Signed-off-by: Treemann <rongzhang.zheng@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Signed-off-by: Treemann <rongzhang.zheng@amd.com>
Signed-off-by: Rongzhang Zheng <rongzhang.zheng@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…roject#441)

* sync: update from Slime through #2393

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix: complete vLLM sync translations

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* sync: complete Slime update through #2394

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* Sync Slime through score centering

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix: rebase vLLM patches onto current nightly

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix: remove stale async CI entry and obsolete metrics fallbacks

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* feat: transport sampling-mask logprobs for score centering

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(vllm): enable scale-out endpoint for rollout TITO

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(rollout): keep requests during sleep and enable external PD TITO

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(ci): enable TITO on external OPD teacher

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* test: enable scale-out render endpoint in Geo3K e2e

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* feat: mirror Slime persistent rollout queue and distributed fully async

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* test: exercise straw transport in mirrored GPU suites

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* style: match repository formatter for straw sync

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* ci: install straw in CPU contract container

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* sync: mirror Slime #2427 straw checkpoint rollback

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* ci: install straw dependency for sync tests

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* test: pin vllm generation config for score centering

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix: bind straw coordinator in vllm rollout

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: record full GPU gate procedure

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* sync: mirror Slime #2432 Straw queue and replay changes

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* ci: quote Straw requirement in Buildkite shell steps

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* [Sync] Mirror Slime PipelineRL and reference top-p fix

Port #2437 flush scheduling and #2434 reference log-prob replay fix. Replace request-bound TITO weight version with per-output propagation through vLLM #53199, and preserve exact top-p score-centering support.

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* [CI] Add flush interval to weight updater test fixtures

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* [CI] Accept versionless terminal abort chunk in PipelineRL probe

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* [Sync] Mirror concurrent rollout and HTTP regression tests

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* [Sync] Preserve buffered-token weight version on abort

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* [Sync] Avoid zero-KL assertion for top-p actor/ref mismatch

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* [Sync] Backport AuxOutput reset fix from vLLM #59060

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* [Sync] Stamp synthetic abort outputs with committed version

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* [Sync] Preserve coalesced rollout metadata and validate sampler probabilities

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* [Sync] Remove superseded per-file spill and mirror Straw references

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* [CI] Inherit the available file descriptor limit for GPU jobs

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* [Sync] Backport terminal token-stream finish reasons

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* [Test] Cover terminal output with real prompt token metadata

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* [Sync] Mirror latest Slime and validate nightly Python/Rust adapters

Mirror Slime through 2f231865 with approved native overlays and Source test registration. Pin validated43b nightly, install tested tensor-dump/tracing Python and Rust patches, and parse native integer-ID top-k responses. Preserve original Straw assertions and saving policy. Keep profiler deferred and latest unchanged.

AI assistance: OpenAI Codex. Scoped validation171CPU tests, full precommit and11patch/98path installation proof pass; full remote CI and remaining candidate GPU validation pending.

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* [Sync] Mirror Slime external PD delta artifact assertion

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix: repair verified recovery and initial-policy CI contracts

Preserve training loss, numerical thresholds and checkpoint strategy. Add indexed precision dumps, offline-first manager setup, scoped reference load, same-distribution CI KL diagnostic, canonical optimizer masters and NIXL peer retirement.

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix: keep initial-model diagnostic optional for direct loss callers

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix: give the PD rollout debug dump a unique per-rollout path

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* [Sync] Mirror Slime v0.4.0 training stack and release updates

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* [Sync] Mirror latest Slime documentation and native engine contracts

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* [Sync] Restore upstream documentation release checks

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(sync): mirror active DeepEP recipes and docs CI contracts

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs(sync): mirror Slime environment entry and preserve sampler parameters

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* [Sync] Preserve upstream export explanation and tokenization attribution

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* [Sync] Inherit Slime documentation deletions and navigation

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* Remove opt-in DeepEP auto switching

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* [RL] Release native weight-transfer communicators and expose lifecycle APIs

Include fork vLLM vllm-project#119 on the fixed 43b4 baseline, including explicit communicator destruction and matching Python/Rust public controls. Keep the other sixteen-patch capabilities intact.

Assisted-by: OpenAI Codex
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* [Sync] Mirror Slime agent and collective updates through 58b08786

Source PRs: THUDM/slime#2453, #2450, #2458, #2457. Preserve approved Vime overlays and restore the omitted Source Straw error assertions. Native TITO, replay/SC, MTP and Buildkite adapters use the fixed 43b4 engine stack.

Validation: Slurm2366 301CPU/1skip,2376 Python3.11 agent step163pass/2optional skips,2377 Straw7+88pass,2378 nine precommit hooks. Native sandbox SSD prerequisites2373 pass; full new8GPU/model and final-image gates remain pending. No latest or release promotion.

Slime-commit: 58b0878679de339ec7844b45d884a87955a134be

Co-authored-by: Codex <noreply@openai.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* [Build] Package native ARM Sunabako without changing x86 installation

Preserve Slime SDK0.1.1 contracts and the original x86 build target. Build a verified native ARM wheel with matching seccomp/syscall/ABI and driver paths before requirements resolution.

Source-Slime: 58b0878679de339ec7844b45d884a87955a134be
Co-authored-by: Codex <noreply@openai.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* Mirror rollout weight-group lifecycle and preserve draft state across sleep

Restore Source connect/disconnect layout and Agent MTP translation. Reuse vLLM parameter retention for draft-private weights instead of adding a second backup switch.

Co-authored-by: Codex <noreply@openai.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* sync: align draft retention patch with reviewed native selector

Mirror the simplified native proposal, including an empty fully-shared draft. No trainer, saving, or inference-path policy change.

Co-authored-by: Codex <noreply@openai.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* [Sync] Mirror router health-check ownership with native gateway patch

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(sync): mirror source documentation and MTP verification budgets

Co-authored-by: Codex <noreply@openai.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(rust): mirror multimodal render-to-generate support

Keep the pinned vLLM base and existing Rust overlays; apply the independent multimodal bridge after them. Preserve explicit UUIDs, native validation and metadata-only EC support. Remove the GB10 launcher resurrected after the approved platform deletion.

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(epd): forward producer transfer metadata and preserve media identity

Use the existing rollout endpoint metadata instead of inert SGLang-only engine keys. Preserve ordinary/streaming placement, complete media parts and caller UUIDs. Do not alias language_only to the unrelated modality-limit flag.

Validation: frozen700-path CPU2502 passes92 cases; real two-host ECCPU/NIXL2503 passes both frontends with four actual transfers and baseline-exact tokens/UUID metadata-only. Explicit connector/body assembly, not default deployment or all-model certification.

Co-authored-by: Codex <noreply@openai.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(epd): mirror encoder deployment with native EC transfer

Keep Slime deployment sequencing and consumer rendering. Use native EC transport and DP endpoint ownership, preserve decoder input processing while omitting encoder weights, and supply resolved engine config and embedding support to Rust. Reuse existing upstream authentication fix; retain test assertions.

Validation: complete 23-patch stack on43b;83CLI/468server and all510Rust paths; installed experimentalfefcf78runtime/705sourcepaths/92CPU; no-overlay original3GPU E-P-D2552 passes Python/Rust. One-hour2554 is running, not claimed complete. Native review worktrees remain uncommitted. AI assistance used.

Co-authored-by: Codex <noreply@openai.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

---------

Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Codex <noreply@openai.com>
…ants (vllm-project#474)

* ci(rocm): ROCm CI test suite + review fixes, rebased onto main

Rebased onto upstream main (which now carries the Slime v0.4.0 sync vllm-project#441:
rebased vLLM patches, enable_scale_out, test_accelerator changes). Drops the
now-redundant docker/patch/vllm_engine content this branch previously copied
from vllm-project#432 — main and vllm-project#432 own those. Keeps only the ROCm CI additions and the
PR #3 review fixes:

- .buildkite/pipeline-rocm.yaml: RAY_EXPERIMENTAL_NOSET_* + NCCL env, the
  HIP->CUDA visibility wrapper (normalizes CUDA_VISIBLE_DEVICES to the locked
  HIP set in-command, keeping tests/ci/gpu_lock_exec.py untouched), and the
  vllm-config/quick-start test steps.
- tests/*-rocm.py: 8 standalone ROCm test variants (self-contained U.is_rocm
  shim; experimental/unwired PPO-critic note; corrected mixed-offload
  level-1-sleep restore description; dead code removed).
- docs amd_tutorial: `sleep infinity` so detached container stays up for exec.

Co-Authored-By: Claude <noreply@anthropic.com>

* ci(rocm): keep green vllm-config suite; bundle validated ROCm image

Retest on the rebuilt image (vllm/vime-rocm:pr-441) cleared startup and the
health gate for all suites; the 4 vllm-config GRPO tests pass to a train step.
Drop the 4 that fail for reasons outside this CI work:
  - quick_start_glm4_9B: GLM-specific step-0 KL exceeds the strict <1e-8 assert
    (ROCm rollout/train logprob divergence; the 0.5B GRPO tests pass the same check)
  - qwen3_4B_ppo{,_train_critic_only,_disaggregate}: SIGSEGV in critic model
    creation (known experimental ROCm PPO-critic path)

Bundle the validated Dockerfile.rocm (layers on vllm/vllm-openai-rocm:nightly,
credit @indianspeedster vllm-project#432) with the fixes the retest surfaced:
  - pin pyarrow==25.0.1: the base's 26.0.0 is a numpy-2 build that rejects the
    numpy 1.26.4 pin at import, breaking Ray serialization + HF datasets
  - apply vllm-generation-health.patch: provides /health_generate, which the
    post-vllm-project#441 rollout health monitor polls (slim 3-patch set had dropped it)
  - drop unused megatron-energon / multi-storage-client
  - CMD sleep infinity so `docker run -d` stays up for exec

Co-Authored-By: Claude <noreply@anthropic.com>

* docker(rocm): isolate ROCm-rebased vLLM patches from the shared CUDA set

d21bad2 overwrote docker/patch/latest/{vllm,vllm-pull_weights,
vllm-inflight-queue-diagnostics}.patch with ROCm-rebased versions, but the CUDA
docker/Dockerfile applies those same files with strict `git apply` — the rebase
would break the CUDA image build.

The ROCm nightly base genuinely needs the rebased patches: main's CUDA versions
apply without rejects but silently misplace the weight_version hunk under --fuzz
(lands on the wrong protocol class), failing the import-check assert.

Resolve by isolation:
  - docker/patch/latest/ reverted to main's versions (CUDA untouched)
  - ROCm-rebased patches moved to docker/amd_patch/latest/ (ROCm-only dir, already
    home to the ROCm megatron patches); Dockerfile.rocm points there
  - vllm-generation-health.patch stays shared (applies clean on both bases)

Rebuilt vllm/vime-rocm:pr-441 from docker/amd_patch/ — import check passes
(weight_version correctly placed). Also trims Dockerfile/pipeline comment blocks
to <=2 lines.

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
@read-the-docs-community

Copy link
Copy Markdown

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the Vime repository by introducing an interactive experiment builder lab to the documentation, restructuring the docs directory, and updating the Docker and Buildkite CI configurations to support new features like speculative decoding, fault tolerance, and straw transport. Feedback on the changes includes installing the missing patch utility in docker/Dockerfile.rocm to prevent build failures and updating a legacy [slime-docs] prefix to [vime-docs] in docs/build.sh for consistency.

Comment thread docker/Dockerfile.rocm
Comment on lines +34 to +36
RUN apt-get update && apt-get install -y \
rsync dnsutils iproute2 prometheus && \
rm -rf /var/lib/apt/lists/*

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The patch utility is used later in the Dockerfile (lines 126-129) to apply various vLLM patches, but it is not installed in this stage. Since the base image vllm/vllm-openai-rocm:nightly may not have patch pre-installed, please add patch to the apt-get install command to prevent potential build failures.

RUN apt-get update && apt-get install -y \
    rsync dnsutils iproute2 prometheus patch && \
    rm -rf /var/lib/apt/lists/*

Comment thread docs/build.sh
Comment on lines +17 to +25
if [ -z "$SPHINX_BUILD" ]; then
cat >&2 <<'EOF'
[slime-docs] sphinx-build was not found. From the repository root, run:
python3 -m venv .venv-docs
.venv-docs/bin/python -m pip install -r docs/requirements.txt
Then retry: bash docs/build_all.sh
EOF
exit 1
fi

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The error message refers to [slime-docs], which is the old name of the project. Since the project has been renamed to Vime, please update this prefix to [vime-docs] to maintain consistency with other build scripts (e.g., docs/build_all.sh uses [vime-docs]).

Suggested change
if [ -z "$SPHINX_BUILD" ]; then
cat >&2 <<'EOF'
[slime-docs] sphinx-build was not found. From the repository root, run:
python3 -m venv .venv-docs
.venv-docs/bin/python -m pip install -r docs/requirements.txt
Then retry: bash docs/build_all.sh
EOF
exit 1
fi
if [ -z "$SPHINX_BUILD" ]; then
cat >&2 <<'EOF'
[vime-docs] sphinx-build was not found. From the repository root, run:
python3 -m venv .venv-docs
.venv-docs/bin/python -m pip install -r docs/requirements.txt
Then retry: bash docs/build_all.sh
EOF
exit 1
fi

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants