Sweeps as throttled Slurm job arrays; the contract's GPU ceiling paces them

[renmengye]

Step 2b of the session-watcher design note (#316): sweeps as throttled Slurm job arrays, replacing the fan-out into N separate jobs.

• One job per sweep. launch --array N submits --array=0-(N-1)%K. The queue holds one entry that accrues priority as one job; Slurm runs at most K tasks at once; the wake's afterany on the array id covers every task. write_eval_job(array=N) writes one script whose task derives its job dir eval-launch-<name>.<k> and SWEEP_INDEX from SLURM_ARRAY_TASK_ID (or SWEEP_INDEX under a local run), and creates and clears the N task dirs. Result gathering by <name>.<k> is unchanged.
• The author sets the pace, the contract caps it. launch --concurrency K (default: the whole array). budgets.max_concurrent_gpus is a ceiling in GPUs: clamp_concurrency sets K to at most ceiling // gpus (a two-GPU task gets half the tasks of a one-GPU task and the same share of the machine), never below one, before the launcher submits and before the park records the request. Clamped, never refused; the wake text says each sweep's pace. Unset ceiling: the author's pace stands.
• Task ids. <id>_<k> is a job id (_check_job_id), so states, elapsed times and cancels work per task. SlurmCompute.status on an array id combines the per-task sacct lines (combine_states: running while any task runs, pending while any waits, terminal only when all are; COMPLETED only if all are). launch_task_ids expands one id per launch for annotate_launch_states and the GPU-hour refund; a park recorded before this change carries one id per task and passes through; an unknown mapping yields none rather than a guess.
• Ledger and queue view. The ledger keeps the array id per launch; the watcher maps <id>_<k> and the pending range <id>_[0-15%4] back to it and reads K off the range, and the tool renders "(sweep, K at a time)".
• Local compute runs an array's tasks in turn, each with SLURM_ARRAY_TASK_ID and its own persisted state; the array's state is theirs combined.
• Docs: contract knob row, design note status, CHANGELOG. Tests across compute, dispatch, local compute, the live author-sleep sweep, request parsing and the clamp, the orchestrator clamp, the watcher's array rows, the tool, the ledger.

Follow-up for the lab: set budgets.max_concurrent_gpus: 12 in the gpt-speedrun contract.

🤖 Generated with Claude Code

[github-actions]

Round 1 — reviewed head fb8aae5e — reviewer summarizer:hermes/gpt-5.6-terra over coverage+credentials+deployment+general+lifecycle+prose.

terra
Advisory findings from outerloop — the code owner decides. Reply to disagree; the outerloop:no-review label opts this PR out.

Verdict: 1 blocking, 2 advisory.

1 finding attached to the lines below.

Submitted sweeps bypass the GPU concurrency ceiling. [deployment+prose] The candidate-pending/park path passes submitted launches directly to launcher without clamp_concurrency, so a submitted array can request full concurrency and exceed budgets.max_concurrent_gpus. (src/outerloop/orchestrator.py:1709; high confidence)

Advisory (non-blocking):

• Suggestion: The changelog entry packs too many ideas into one sentence. [prose] Sweeps now use Slurm job arrays. Use --concurrency to limit active tasks. max_concurrent_gpus caps that limit. Local compute runs array tasks in order. The queue shows the limit. (CHANGELOG.md:91; high confidence)

Verdict: blocking—submitted sweep arrays bypass the GPU concurrency clamp (deployment and prose agree); additionally, the queue can lose a running sweep’s configured pace, and the changelog needs the supplied prose rewrite. Rejected: none; the prose orchestrator finding was consolidated with the equivalent deployment finding rather than recorded separately.

[github-actions]

Round 1 — reviewed head b9c4e3a9 — reviewer hermes/gpt-5.6-terra.

terra
Advisory findings from outerloop — the code owner decides. Reply to disagree; the outerloop:no-review label opts this PR out.

Verdict: no defects found.