feat(power): multinode measured-power aggregation

[arygupt]

Summary

Extends single-node measured-power aggregation (#1558) to multinode srt-slurm benchmarks. Wires per-node perf_samples_<host>.csv from srt-slurm's PR #35 perfmon through the launcher into process_result.py → aggregate_power.py, which now namespaces local GPU indices per source CSV stem so each node's local indices 0..N-1 don't collapse across nodes.

Backward compatible: aggregate_power() accepts both Path and Iterable[Path]; single-CSV callers (single-node start_gpu_monitor path) are unchanged. csv_path param name preserved.

Pipeline

srt-slurm perfmon (PR #35 on NVIDIA/srt-slurm, layered on
  SemiAnalysisAI/srt-slurm:feat/inferencex-perfmon)
  → perf_samples_<host>.csv in outputs/<job>/logs/ on shared NFS
  → launch_gb300-cw.sh exports GPU_METRICS_CSV_GLOB to $GITHUB_ENV
  → process_result.py expands glob → aggregate_power.run() with list
  → aggregate_power.py emits cluster-wide avg_power_w + joules_per_*_token
  → InferenceX-app ETL auto-captures (no schema change)

Files

• utils/aggregate_power.py — csv_path widened to Path | Iterable[Path]. Per-source GPU-id namespacing only kicks in for 2+ sources so single-node num_gpus is unchanged. CLI adds --csv-glob (mutually exclusive with --csv).
• utils/process_result.py — bridge GPU_METRICS_CSV_GLOB env var. Glob takes precedence over single GPU_METRICS_CSV when both are set.
• runners/launch_gb300-cw.sh — point dynamo-sglang at our srt-slurm fork, append monitoring: block to each recipe post-copy (idempotent), write GPU_METRICS_CSV_GLOB to $GITHUB_ENV after the job.
• utils/test_aggregate_power.py — 8 new multinode cases: per-source namespacing, sub-second clock drift, asymmetric prefill/decode power, missing-CSV silent skip, backward-compat single-path-in-list, Iterable acceptance, E2E with list.
• utils/test_process_result.py — 3 new cases: glob aggregation, precedence over single CSV, empty-match falls through.

Test plan

• 36/36 aggregator tests pass (28 existing + 8 new)
• 28/28 process_result tests pass (25 existing + 3 new)
• nvidia-smi inside sglang container on real gb300-cw emits expected columns (timestamp, gpu, power_w) — verified manually with srun --container-image=...sglang...sqsh nvidia-smi --query-gpu=...
• First E2E multinode sweep produces avg_power_w + joules_per_*_token in the agg JSON (pending perf-changelog.yaml entry + sweep-enabled label)
• num_gpus in agg JSON matches prefill_gpus + decode_gpus from launcher (validates per-source namespacing — without the fix, num_gpus would equal a single node's gpus_per_node)
• Chart at inferencex.semianalysis.com renders the new data via ?unofficialrun=<run_id>

Depends on

SemiAnalysisAI/srt-slurm:feat/inferencex-perfmon (pinned by the launcher). Tracks NVIDIA/srt-slurm PR #35 head; will rebase to upstream main once #35 merges.

---

Note

Medium Risk
Changes benchmark CI/launcher behavior and published agg JSON schema (new power/telemetry fields); failures are mostly best-effort for monitoring, but incorrect glob or GPU namespacing could silently skew power metrics until caught by smoke tests.

Overview
Extends measured-power from single-node to multinode by collecting per-node perf_samples_<role>_w<idx>_<host>.csv files and patching agg JSONs with cluster-wide and (when disagg) per-stage energy metrics.

NVIDIA (gb300 / srt-slurm): launch_gb300-cw.sh pins a fork with per-node perfmon, recursively injects monitoring: enabled into overlaid recipe YAMLs (fixes silent zero-power when recipes live in subdirs), exports GPU_METRICS_CSV_GLOB after the job. AMD (mi355x): each SGLang/vLLM node starts start_perf_monitor into NFS-shared perfmon/; launch_mi355x-amds.sh copies CSVs to the workspace before log cleanup and sets the same glob env var.

Aggregation pipeline: process_result.py prefers GPU_METRICS_CSV_GLOB over single GPU_METRICS_CSV (no stale single-node fallback when glob is set) and passes disagg into aggregate_power.run(). aggregate_power.py now accepts multiple CSVs with per-source GPU index namespacing, parses perfmon filenames for workers[], adds temp/util/mem/peak fields, disagg per-stage joules_per_input_token / decode-weighted joules_per_output_token, and bench-window fallbacks for srt-slurm date + duration when Unix timestamps are absent.

Ops hardening: Multinode workflow Slurm pre-cleanup uses timeouts, drain deadlines, and AMD-only sudo rm of stale benchmark_logs/; AMD launcher job_alive() tolerates squeue timeouts. AMD monitoring adds -m for VRAM and logs CSV headers for schema-debug.

Large test additions cover multinode/disagg/AMD CSV shapes and process_result glob behavior; perf-changelog.yaml documents smoke sweeps for validation.

^{Reviewed by Cursor Bugbot for commit 60b62ed. Bugbot is set up for automated code reviews on this repo. Configure here.}

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26534043069
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=26534043069

[claude]

🔴 When GPU_METRICS_CSV_GLOB is set but matches no files, _csv_arg stays None and the code falls through to the single-CSV candidate list (GPU_METRICS_CSV, gpu_metrics.csv, /workspace/gpu_metrics.csv) — contradicting the comment at lines 145-148 that the glob 'Takes precedence over the single-CSV fallback'. On a persistent self-hosted runner with a stale /workspace/gpu_metrics.csv from a prior single-node run (or a leaked GPU_METRICS_CSV env var), a multinode run whose perfmon failed on every node would silently patch wrong single-node avg_power_w / joules_per_*_token values into the multinode agg JSON. Fix: when _glob_pattern is truthy, skip the single-CSV fallback regardless of whether the glob matched anything.

Extended reasoning...

The contract violation

The block at utils/process_result.py:142-159 documents the precedence contract clearly:

Takes precedence over the single-CSV fallback — if the launcher set the glob, the run was multinode and there is no single-CSV fallback to make.

But the implementation only honors that contract when the glob actually matches files. On empty match:

_csv_arg = None
_glob_pattern = os.environ.get('GPU_METRICS_CSV_GLOB')
if _glob_pattern:
    _matched = sorted(Path(p) for p in _glob_module.glob(_glob_pattern))
    if _matched:
        _csv_arg = _matched
    else:
        print(..., file=sys.stderr)   # warns but doesn't prevent fallthrough

if _csv_arg is None:                  # still None — falls into single-CSV branch
    _csv_candidates = [
        os.environ.get('GPU_METRICS_CSV'),
        'gpu_metrics.csv',
        '/workspace/gpu_metrics.csv',
    ]
    ...

The else branch just logs; _csv_arg stays None, and the next if _csv_arg is None block consults the single-CSV candidates.

Step-by-step proof on a persistent self-hosted runner

1. Single-node run on gb300-cw_N completes successfully. benchmarks/benchmark_lib.sh exports GPU_METRICS_CSV=/workspace/gpu_metrics.csv (it lives in gpu_metrics.csv in cwd too). The file is left behind because the runner is persistent across jobs.
2. Next job is a multinode dynamo-sglang sweep. runners/launch_gb300-cw.sh (lines 297-318) writes GPU_METRICS_CSV_GLOB=$LOGS_DIR/perf_samples_*.csv to $GITHUB_ENV — but only when perf_csv_count > 0. Suppose perfmon failed to start on every node (srt-slurm PR [NVIDIA] Reduce B200 Runs & add B200 FP4 Docker Script #35 had startup issues, host driver mismatch, etc.) — perf_csv_count would be 0 and the glob env var would not be written. Fine — that path is safe.
3. However, suppose perfmon CSVs were written at the end of the job (so the launcher writes the GLOB), but a downstream cleanup hook between launcher and process_result.py removed them, OR srt-slurm wrote the CSVs to a different path on a subsequent retry, OR a persistent env var (GPU_METRICS_CSV_GLOB from a prior job) leaks in. The glob expansion in process_result.py returns empty.
4. process_result.py enters the else branch on line 155, prints a warning, and falls through. os.environ.get('GPU_METRICS_CSV') from the prior single-node job returns /workspace/gpu_metrics.csv (or gpu_metrics.csv in cwd is still there). Path(p).is_file() is True. _csv_arg = Path('/workspace/gpu_metrics.csv').
5. _aggregate_power_run is called with the stale single-node CSV.

Why the bench-window timestamp filter doesn't always save us

One verifier argued the start_unix <= ts <= end_unix filter at aggregate_power.py:177-178 would reject stale samples. That's true if the window comes from explicit Unix timestamps. But this PR adds two new fallback tiers in _load_bench_window:

• Tier 2: date field parsed as a UTC string (YYYYMMDD-HHMMSS).
• Tier 3: bench_result_path.stat().st_mtime — the bench JSON's own mtime, which is the current run's mtime, used as bench-end with start = end - duration.

The mtime tier is exactly the danger zone: on a persistent runner the bench JSON is freshly written, so its mtime is now. If the stale gpu_metrics.csv was also written recently (within the derived [mtime - duration, mtime] window — possible if the prior single-node run finished a few minutes ago), its samples do fall inside the window. Result: silent wrong avg_power_w and joules_per_*_token patched into the multinode agg JSON, which InferenceX-app's ETL auto-captures into the dashboard.

What the test misses

The accompanying test test_multinode_csv_glob_empty_match_falls_through_silently only asserts the no-stale-file case (asserts 'avg_power_w' not in patched). It does not stage a stale fallback CSV, so it can't catch the precedence violation. test_multinode_csv_glob_takes_precedence_over_single_csv only tests precedence when the glob matches.

Fix

One-line change in the empty-match branch:

if _glob_pattern:
    _matched = sorted(Path(p) for p in _glob_module.glob(_glob_pattern))
    if _matched:
        _csv_arg = _matched
    else:
        _csv_arg = []           # sentinel: glob attempted, fallback forbidden
        print(...)

if not _csv_arg:                # treats [] same as None for the downstream check, but…
    if _glob_pattern:
        pass                    # …skip single-CSV candidates when glob was attempted
    else:
        _csv_candidates = [...]
        _csv_arg = next(...)

Or more cleanly: guard the single-CSV block on not _glob_pattern instead of _csv_arg is None.

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26547958720
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=26547958720

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26548110246
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=26548110246

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26593269421
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=26593269421

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26600823211
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=26600823211

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit d2cca8a. Configure here.}

[cursor]

Squeue failure stalls job wait

Medium Severity

The new job_alive helper treats any failed squeue as proof the job is still running. After the Slurm job has already finished, repeated controller timeouts leave the launcher in the background poll and log-file wait loops indefinitely instead of proceeding to perfmon collection and result processing.

 

^{Reviewed by Cursor Bugbot for commit d2cca8a. Configure here.}

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27037497545
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27037497545

[arygupt]

@Oseltamivir (or anyone on the Core team, @SemiAnalysisAI/core) — could I get a review on this when you have a cycle? 🙏 It's the gate on the measured-power work landing.

What it is: measured-power + temperature aggregation for multinode runs — turns the disagg energy numbers from modeled into measured by reading per-node perfmon CSVs (srt-slurm on NVIDIA, amd-smi on AMD) and patching the agg JSON. Purely additive telemetry; no production config behavior changes.

Where to focus review:

• utils/aggregate_power.py — per-source GPU-id namespacing (a TP16 worker over 2 nodes counts 16 GPUs, not 8), per-worker workers[] with role labels (prefill/decode/agg/frontend), per-stage prefill_avg_power_w/decode_avg_power_w + joules_per_*_token, and per-worker avg_temp_c/peak_temp_c/avg_util_pct/avg_mem_used_mb.
• utils/process_result.py — the GPU_METRICS_CSV_GLOB bridge into the Process-result step.
• Launchers — launch_mi355x-amds.sh, launch_gb300-cw.sh, amd_utils/*, benchmark_lib.sh: CSV collection + monitoring: wiring.
• Tests — test_aggregate_power.py, test_process_result.py cover the new fields.

Validation: the changelog entries (dsv4-gb300 + 6 AMD disagg) are smoke runs that produce the first agg JSONs with avg_power_w + per-worker power/temp populated — that's what the in-flight sweep (run 27037497545) is exercising.

Happy to walk through any of it — thanks!