Add spur (Crusoe) scheduler support for multi-node runs#157
Draft
mkuznet1 wants to merge 7 commits into
Draft
Conversation
Adapt the SLURM deployment path for the spur scheduler (slurm-compat shim,
v0.4.1) whose CLI is a subset of stock SLURM and whose control plane is
Raft-based / eventually consistent. These changes keep stock SLURM behavior
intact (guarded by feature/format detection) while making the same code work
on spur.
- Nodelist expansion: add _expand_nodelist() (Python) and mad_expand_nodelist()
(job.sh.j2) helpers. Stock SLURM emits a compressed nodelist and needs
`scontrol show hostnames`; spur exposes an already-expanded comma list and
does not implement `scontrol show hostname[s]`. Prefer comma-split for the
expanded form, fall back to scontrol only for compressed "node[..]" forms.
MASTER_ADDR, sglang-disagg node IPs, and the deepspeed hostfile now use this.
- Node rank: derive SGLANG_NODE_RANK from SLURM_PROCID, then NODE_RANK /
PMI_RANK / SLURM_NODEID, since spur leaves SLURM_PROCID empty inside srun.
- Submission: retry sbatch on transient Raft errors ("not the Raft leader",
"no leader elected yet", "service is currently unavailable").
- Completion check: drop unsupported `sacct -X`; parse the first (main job)
row; treat active states (RUNNING/PENDING/...) as RUNNING instead of FAILED
so a transient empty squeue result does not mark a live job as failed.
Co-authored-by: Cursor <cursoragent@cursor.com>
spur's srun cannot dispatch tasks to other nodes (runs once on the head node,
SLURM_PROCID empty), which breaks the standard `srun bash task` per-node model.
Add a spur deployment target that drives multi-node execution via a job ARRAY
of single-node tasks instead.
- New SpurDeployment(SlurmDeployment) target ("spur"), registered in the
deployment factory. It reuses the SLURM template/flow and only injects
scheduler="spur" + a shared-filesystem rendezvous_dir into the template context.
- job.sh.j2 gains {% if scheduler == 'spur' %} branches:
* header uses --nodes=1 --ntasks=1 --array=0-(N-1) instead of a multi-node job;
* node rank = SLURM_ARRAY_TASK_ID; MASTER_ADDR is resolved via a shared-FS
rendezvous (rank 0 publishes its ens3 IP, peers read it), matching the model
launcher's TCP rendezvous; port keyed off SLURM_ARRAY_JOB_ID;
* the per-node task body runs directly (bash) instead of via `srun`.
- Stock SLURM behavior is unchanged (scheduler defaults to "slurm").
Co-authored-by: Cursor <cursoragent@cursor.com>
spur's `sacct -j <id>` ignores the id filter and returns the whole cluster accounting history, so per-job state cannot be read from it (this is what made job states appear to "flap"). Make the spur backend robust: - SpurDeployment.monitor(): detect completion from per-rank marker files on the shared filesystem (<rendezvous_dir>/<array_job_id>/done_rank<N> = exit code), written by each array task. squeue (filtered by job name) is used only as a liveness guard so monitoring cannot hang if a task dies without a marker. - job.sh.j2: each spur array task writes its done_rank<N> marker; per-node logs and the node collection dir are now keyed by MAD_COLLECT_JOB_ID (= SLURM_ARRAY_JOB_ID on spur, = SLURM_JOB_ID on stock SLURM) so the existing collect_results(deployment_id=array_job_id) finds them. Stock SLURM behavior is unchanged. Co-authored-by: Cursor <cursoragent@cursor.com>
- run_orchestrator: an explicit non-local deployment_config.target now takes precedence over structural inference. The spur backend reuses the "slurm" config block, so inference alone mis-detected "spur" manifests as "slurm". - job.sh.j2 (spur branch): pin SLURM_JOB_ID to SLURM_ARRAY_JOB_ID for the container so the model launcher's rendezvous port and shared /run_logs/<id> dir are identical on every node (array tasks otherwise each have a distinct SLURM_JOB_ID). The per-node task script filename now includes the node rank to stay unique on the shared filesystem, and SLURM_SUBMIT_DIR is defaulted. Co-authored-by: Cursor <cursoragent@cursor.com>
On spur, squeue does not list freshly-submitted array tasks for ~1-2 min (registration lag / eventual consistency), so a healthy run reports 0 live tasks at startup. The previous liveness guard counted those startup polls and aborted the run as failed, orphaning the still-starting tasks. Only start the "died without a marker" countdown after the tasks have been observed alive at least once, and treat squeue-unavailable (unknown) polls as non-fatal. Co-authored-by: Cursor <cursoragent@cursor.com>
spur's squeue renders `-o "%j|%T"` as space-separated "<name> <state>",
ignoring the literal delimiter, so the previous partition("|") parse always
yielded 0 live tasks. Request "%j %T" and split on whitespace instead.
Co-authored-by: Cursor <cursoragent@cursor.com>
Phase B / portable fallback: the self-managed "slurm_multi" launcher path also relied on srun to fan out (parallel docker pull with `srun --nodes=$SLURM_NNODES` and the model script's own internal srun), which does not work on spur. Add an IS_SPUR flag (False on SlurmDeployment, True on SpurDeployment) and spur-conditional branches in _prepare_slurm_multi_script: - header submits a job array (--nodes=1 --ntasks=1 --array=0-(N-1)); - each array task derives NODE_RANK from SLURM_ARRAY_TASK_ID, pins SLURM_JOB_ID to the shared SLURM_ARRAY_JOB_ID, resolves MASTER_ADDR via the shared-FS rendezvous, and pulls the image locally (no srun); - each task writes a done_rank<N> marker consumed by SpurDeployment.monitor(). Stock SLURM slurm_multi generation is unchanged (verified: nodes=N/ntasks=N, srun pull, no array/rendezvous/markers). Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds support for the spur (Crusoe) scheduler, whose SLURM-compatible CLI
breaks stock multi-node fan-out (
srundoes not dispatch tasks to othernodes, empty
SLURM_PROCID;scontrol show hostnamesunsupported; Raft-basedeventually-consistent squeue/sacct).
SpurDeploymentbackend (target: spur) that fans out multi-node workvia a job array (one single-node task per node);
SLURM_ARRAY_TASK_IDis the node rank. Nodes self-form the cluster through the launcher's TCP
rendezvous (rank 0 publishes its transport IP to a shared FS, peers read it
as
MASTER_ADDR).monitor()(per-rankdone_rank<N>files) because spur
sacct -jdoes not filter by job id; squeue used onlyas an eventually-consistent liveness guard.
job.sh.j2: all spur logic gated behind{% if scheduler == 'spur' %};MAD_COLLECT_JOB_IDunifies log/artifact paths (==SLURM_JOB_IDon stockSLURM,
SLURM_ARRAY_JOB_IDon spur).targetnow wins over structuralinference (spur reuses the
slurmconfig block).slurm_multilauncher path reworked to fan out via job array on spur.Stock SLURM behavior is unchanged (spur paths are fully gated; render + import
smoke-checked).
Test plan
slurmandspurschedulersunrelated docker image build issue)