Skip to content

[codex] Remove MI325X runner workarounds#1747

Merged
cquil11 merged 1 commit into
mainfrom
codex/remove-mi325x-runner-workarounds
Jun 13, 2026
Merged

[codex] Remove MI325X runner workarounds#1747
cquil11 merged 1 commit into
mainfrom
codex/remove-mi325x-runner-workarounds

Conversation

@cquil11

@cquil11 cquil11 commented Jun 13, 2026

Copy link
Copy Markdown
Collaborator
  • Remove stale benchmark log cleanup from the MI325X AMD launcher.
  • Allow Slurm to schedule chi-mi325x-pod1-121.ord.vultr.cpe.ice.amd.com.

Note

Low Risk
Launcher-only script changes with no auth or data handling; main risk is jobs landing on a previously excluded node if the enroot/NFS issue regresses.

Overview
Removes temporary mitigations from runners/launch_mi325x-amds.sh now that the underlying issues are addressed.

Slurm scheduling: salloc no longer passes --exclude for chi-mi325x-pod1-121, so that node can receive jobs again.

Cleanup: Deletes cleanup_stale_benchmark_logs and all invocations (startup and on EXIT), including the benchmark_logs removal under GITHUB_WORKSPACE. The exit trap only cancels the job and preserves the exit code.

Reviewed by Cursor Bugbot for commit 0f0d7cb. Bugbot is set up for automated code reviews on this repo. Configure here.

@cquil11 cquil11 marked this pull request as ready for review June 13, 2026 23:40
@cquil11 cquil11 requested a review from a team June 13, 2026 23:40
@cquil11 cquil11 merged commit b7740fb into main Jun 13, 2026
3 checks passed
@cquil11 cquil11 deleted the codex/remove-mi325x-runner-workarounds branch June 13, 2026 23:40
@claude

claude Bot commented Jun 13, 2026

Copy link
Copy Markdown
Contributor

Claude finished @cquil11's task in 47s —— View job


PR Review: Remove MI325X runner workarounds

  • Gather context and review changes
  • Validate against review guidelines
  • Post review

LGTM — no blocking issues found.

This is a clean removal of two workarounds that are no longer needed:

  1. Removed cleanup_stale_benchmark_logs function and its calls — eliminates the sudo rm -rf / rm -rf cleanup of $GITHUB_WORKSPACE/benchmark_logs from both the startup path and the EXIT trap.
  2. Removed --exclude=chi-mi325x-pod1-121.ord.vultr.cpe.ice.amd.com from salloc — re-enables Slurm scheduling on this node now that the underlying enroot/aufs issue (originally tracked via [Klaud Cold] Update gptoss-fp4-mi325x-vllm vLLM ROCm image to v0.21.0 #1467/[Klaud Cold] Update kimik2.5-int4-mi325x-vllm vLLM ROCm image to v0.21.0 #1468/[Klaud Cold] Update minimaxm2.5-fp8-mi325x-vllm vLLM ROCm image to v0.21.0 #1469) has been resolved.

The resulting script is simpler and the changes are straightforward with no functional risk.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Development

Successfully merging this pull request may close these issues.

1 participant