[AMD] dsv4-fp4-mi355x-atom: enable DPA at high concurrency, update image to atom0.1.4 by seungrokj · Pull Request #1717 · SemiAnalysisAI/InferenceX

seungrokj · 2026-06-12T12:41:18Z

Summary

Update image to rocm/atom:rocm7.2.4_ubuntu24.04_py3.12_pytorch_release_2.10.0_atom0.1.4_20260612
Enable on top of DPA+TP8 at high concurrency:
- ISL=1024 / OSL=1024
- ISL=8192 / OSL=1024
Update ISL=8192 search-space: TP8-only conc=4–64, DPA conc=128–1024 (previously conc=1–64 and DPA conc=64–512)
Add perf-changelog entry

Motivation

Based on Pareto frontier analysis from ATOM runs 27367309656 and 27030375093,

Test plan

Verify bash -n syntax check passes on dsv4_fp4_mi355x_atom.sh

🤖 Generated with Claude Code

Note

Medium Risk
Changes benchmark matrix size and ATOM serving flags (DPA/TBO, memory, caching) for a large model on MI355X, which can affect sweep cost, stability, and comparability to prior runs—not app security, but meaningful for CI/perf regression signal.

Overview
Updates DeepSeek-V4 FP4 / MI355X / ATOM benchmark coverage and how the runner starts the server.

dsv4-fp4-mi355x-atom in amd-master.yaml switches the container to rocm/atom-dev:nightly_202606161823 and drops the old day-0 “CONC=1 only” comment block. For 1K/1K, DPA sweeps now run through conc 2048 (was capped at 1024). For 8K/1K, the search space is reworked: explicit TP4 and TP8 conc-list low-concurrency points replace the prior TP8-only conc 1–64 + DPA 64–512 layout, and DPA runs from conc 128 through 2048.

dsv4_fp4_mi355x_atom.sh turns on --enable-tbo with GPU_MAX_HW_QUEUES=5 when DPA is enabled at high concurrency (1K/1K ≥1024, 8K/1K ≥256). Server launch adds CUDA graph capture sizes, --gpu-memory-utilization 0.9, --no-enable_prefix_caching, and use_index_cache / index_topk_freq HF overrides. EVAL_ONLY now sets EVAL_MAX_MODEL_LEN via compute_eval_context_length.

perf-changelog.yaml documents the config change (note: changelog text references a different atom0.1.4 image tag than the YAML pin in this diff).

^{Reviewed by Cursor Bugbot for commit bcf0d1f. Bugbot is set up for automated code reviews on this repo. Configure here.}

…e image to atom0.1.4 - Enable --enable-tbo for ISL=1024/OSL=1024 at CONC>=1024 and ISL=8192/OSL=1024 at CONC>=256 - Update image to atom0.1.4_20260612 - Update ISL=8192 search-space to start at conc=4 and use DPA from conc=128 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

github-actions · 2026-06-12T12:41:28Z

Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook

If it is not, please create a PR first before we can merge your single node PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. If re-running failed jobs is attempted, PR authors are responsible for ensuring it passes. See GitHub's docs on re-running failed jobs: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow

As a rule of thumb, generally, PR authors should request a review & get a PR approval from the respective companies' CODEOWNERS before requesting a review from core maintainers.

If additional help is needed, PR authors can reach out to core maintainers over Slack.

github-actions · 2026-06-12T12:41:28Z

Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook

If it is not, please create a PR first before we can merge your single node PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. If re-running failed jobs is attempted, PR authors are responsible for ensuring it passes. See GitHub's docs on re-running failed jobs: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow

As a rule of thumb, generally, PR authors should request a review & get a PR approval from the respective companies' CODEOWNERS before requesting a review from core maintainers.

If additional help is needed, PR authors can reach out to core maintainers over Slack.

github-actions · 2026-06-12T12:41:28Z

Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook

If it is not, please create a PR first before we can merge your single node PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. If re-running failed jobs is attempted, PR authors are responsible for ensuring it passes. See GitHub's docs on re-running failed jobs: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow

As a rule of thumb, generally, PR authors should request a review & get a PR approval from the respective companies' CODEOWNERS before requesting a review from core maintainers.

If additional help is needed, PR authors can reach out to core maintainers over Slack.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

github-actions · 2026-06-12T12:59:36Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27416363770
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27416363770

…onc range - Pass --max-model-len to server using SERVE_MAX_MODEL_LEN - Add EVAL_ONLY path: compute eval context length via compute_eval_context_length - Extend conc-end to 8192 (isl=1024) and 4096 (isl=8192) in amd-master.yaml Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

github-actions · 2026-06-12T13:18:19Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27417198968
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27417198968

github-actions · 2026-06-12T14:23:22Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27418122192
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27418122192

github-actions · 2026-06-12T17:08:40Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27418122192
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27418122192

…sable max-model-len Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…m-seqs Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

functionstackx · 2026-06-13T16:20:20Z

can u rebase it so the sweep triggers?

…e, enable max-model-len Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

github-actions · 2026-06-17T05:57:19Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27668923188
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27668923188

github-actions · 2026-06-17T05:58:57Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27668950324
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27668950324

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

github-actions · 2026-06-17T06:01:45Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27669004948
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27669004948

github-actions · 2026-06-17T08:20:34Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27669101609
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27669101609

…nch override Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

github-actions · 2026-06-17T08:25:14Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27675672700
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27675672700

…/TP8/DPA Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

github-actions · 2026-06-17T11:21:40Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27676739575
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27676739575

github-actions · 2026-06-17T11:56:21Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27676739575
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27676739575

seungrokj · 2026-06-17T13:02:17Z

@functionstackx @cquil11 can you please merge this ?

chunfangamd

LGTM

seungrokj · 2026-06-17T13:06:23Z

/reuse-sweep-run

seungrokj · 2026-06-17T13:06:26Z

/merge-prs

Oseltamivir

lgtm

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 471aff2. Configure here.}

# Conflicts: # perf-changelog.yaml

* [AMD] dsv4-fp4-mi355x-atom: enable DPA TBO at high concurrency, update image to atom0.1.4 - Enable --enable-tbo for ISL=1024/OSL=1024 at CONC>=1024 and ISL=8192/OSL=1024 at CONC>=256 - Update image to atom0.1.4_20260612 - Update ISL=8192 search-space to start at conc=4 and use DPA from conc=128 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [AMD] perf-changelog: dsv4-fp4-mi355x-atom DPA TBO + image atom0.1.4 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [AMD] perf-changelog: add PR link #1717 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [AMD] dsv4_fp4_mi355x_atom.sh: disable prefix caching Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [AMD] dsv4-fp4-mi355x-atom: add max-model-len, eval context, extend conc range - Pass --max-model-len to server using SERVE_MAX_MODEL_LEN - Add EVAL_ONLY path: compute eval context length via compute_eval_context_length - Extend conc-end to 8192 (isl=1024) and 4096 (isl=8192) in amd-master.yaml Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [AMD] dsv4-fp4-mi355x-atom: narrow eval to single conc=1024 point, disable max-model-len Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [AMD] dsv4_fp4_mi355x_atom.sh: add cudagraph-capture-sizes and max-num-seqs Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [AMD] dsv4-fp4-mi355x-atom: bump to nightly image, expand search space, enable max-model-len Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [AMD] set GPU_MAX_HW_QUEUES=5 in dsv4_fp4_mi355x_atom.sh Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [AMD] dsv4-fp4-mi355x-atom: disable TBO, add TP4 rows for isl=8192, cap conc ranges Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [AMD] dsv4_fp4_mi355x_atom.sh: quote SERVER_LOG variable Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [AMD] dsv4_fp4_mi355x_atom.sh: comment out dense cudagraph sizes Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [AMD] dsv4_fp4_mi355x_atom.sh: fix --hf-overrides JSON escaping Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [AMD] dsv4_fp4_mi355x_atom.sh: comment out dense cudagraph sizes Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [AMD] dsv4-fp4-mi355x-atom: expand search space, restore isl=1024 rows Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [AMD] perf-changelog: update dsv4-fp4-mi355x-atom image and search-space description Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [AMD] dsv4_fp4_mi355x_atom.sh: restore sparse cudagraph capture sizes Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [AMD] perf-changelog: revert dsv4-fp4-mi355x-atom image/search-space, remove stale entries Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [AMD] perf-changelog: add dsv4-fp4-mi355x-sglang entry for PR #1762 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * update dsv4-fp4-mi355x-atom: bump image, enable TBO conditionally, fix mem frac Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * expand dsv4-fp4-mi355x-atom search space: restore ISL1024 scenarios, add TP4/TP8 conc lists for ISL8192 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Update perf-changelog.yaml * Update perf-changelog.yaml * Update perf-changelog.yaml * Update perf-changelog.yaml * update perf-changelog: move dsv4-fp4-mi355x-atom entry to end Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * narrow dsv4-fp4-mi355x-atom to DPA conc=256-2048 ISL8192, fix TBO branch override Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * restore full dsv4-fp4-mi355x-atom search space: ISL1024 + ISL8192 TP4/TP8/DPA Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * chore: retrigger dsv4 atom benchmark sweep --------- Co-authored-by: seungrokj <seungrok.jung@amd.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: seungrokj <144636725+seungrokj@users.noreply.github.com>

seungrokj and others added 2 commits June 12, 2026 21:38

[AMD] perf-changelog: dsv4-fp4-mi355x-atom DPA TBO + image atom0.1.4

c566e28

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

seungrokj requested a review from a team June 12, 2026 12:41

seungrokj requested review from 1am9trash, billishyahao, chunfangamd and yctseng0211 as code owners June 12, 2026 12:41

github-project-automation Bot added this to InferenceMAX Board Jun 12, 2026

[AMD] perf-changelog: add PR link #1717

7e1aa06

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

seungrokj added AMD full-sweep-enabled labels Jun 12, 2026

[AMD] dsv4_fp4_mi355x_atom.sh: disable prefix caching

65e0fa3

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

seungrokj removed the full-sweep-enabled label Jun 13, 2026

[AMD] dsv4-fp4-mi355x-atom: narrow eval to single conc=1024 point, di…

c3b3289

…sable max-model-len Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

cursor Bot reviewed Jun 13, 2026

View reviewed changes

Comment thread .github/configs/amd-master.yaml Outdated

Comment thread benchmarks/single_node/fixed_seq_len/dsv4_fp4_mi355x_atom.sh Outdated

[AMD] dsv4_fp4_mi355x_atom.sh: add cudagraph-capture-sizes and max-nu…

7ffa976

…m-seqs Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

cursor Bot reviewed Jun 13, 2026

View reviewed changes

Comment thread benchmarks/single_node/fixed_seq_len/dsv4_fp4_mi355x_atom.sh

[AMD] dsv4-fp4-mi355x-atom: bump to nightly image, expand search spac…

f2677b2

…e, enable max-model-len Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

cursor Bot reviewed Jun 15, 2026

View reviewed changes

Comment thread .github/configs/amd-master.yaml Outdated

[AMD] set GPU_MAX_HW_QUEUES=5 in dsv4_fp4_mi355x_atom.sh

f5f0d66

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

seungrokj changed the title ~~[AMD] dsv4-fp4-mi355x-atom: enable DPA TBO at high concurrency, update image to atom0.1.4~~ [AMD] dsv4-fp4-mi355x-atom: enable DPA at high concurrency, update image to atom0.1.4 Jun 15, 2026

Update perf-changelog.yaml

5fbd068

update perf-changelog: move dsv4-fp4-mi355x-atom entry to end

d080faa

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

narrow dsv4-fp4-mi355x-atom to DPA conc=256-2048 ISL8192, fix TBO bra…

91f6277

…nch override Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

seungrokj removed the full-sweep-enabled label Jun 17, 2026

restore full dsv4-fp4-mi355x-atom search space: ISL1024 + ISL8192 TP4…

4364ef9

…/TP8/DPA Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

seungrokj added the full-sweep-enabled label Jun 17, 2026

chunfangamd approved these changes Jun 17, 2026

View reviewed changes

Oseltamivir approved these changes Jun 18, 2026

View reviewed changes

seungrokj and others added 2 commits June 18, 2026 09:38

Update perf-changelog.yaml

6644109

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Update perf-changelog.yaml

471aff2

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

cursor Bot reviewed Jun 18, 2026

View reviewed changes

Comment thread perf-changelog.yaml

Oseltamivir added 2 commits June 18, 2026 08:49

fix: resolve PR 1717 changelog conflict

67b052f

Merge remote-tracking branch 'origin/main' into amd/dsv4_atom_0612

bcf0d1f

# Conflicts: # perf-changelog.yaml

seungrokj merged commit 60bf726 into main Jun 18, 2026
26 checks passed

seungrokj deleted the amd/dsv4_atom_0612 branch June 18, 2026 00:56

github-project-automation Bot moved this to Done in InferenceMAX Board Jun 18, 2026

Oseltamivir mentioned this pull request Jun 18, 2026

[AMD] retrigger dsv4-fp4-mi355x-atom benchmark sweep #1817

Merged

Oseltamivir restored the amd/dsv4_atom_0612 branch June 18, 2026 01:19

Oseltamivir mentioned this pull request Jun 19, 2026

[codex] Validate changelog matrix before sweep reuse #1853

Merged

Uh oh!

Conversation

seungrokj commented Jun 12, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Motivation

Test plan

Uh oh!

github-actions Bot commented Jun 12, 2026

Uh oh!

github-actions Bot commented Jun 12, 2026

Uh oh!

github-actions Bot commented Jun 12, 2026

Uh oh!

github-actions Bot commented Jun 12, 2026

Uh oh!

github-actions Bot commented Jun 12, 2026

Uh oh!

github-actions Bot commented Jun 12, 2026

Uh oh!

github-actions Bot commented Jun 12, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

functionstackx commented Jun 13, 2026

Uh oh!

Uh oh!

github-actions Bot commented Jun 17, 2026

Uh oh!

github-actions Bot commented Jun 17, 2026

Uh oh!

github-actions Bot commented Jun 17, 2026

Uh oh!

github-actions Bot commented Jun 17, 2026

Uh oh!

github-actions Bot commented Jun 17, 2026

Uh oh!

github-actions Bot commented Jun 17, 2026

Uh oh!

github-actions Bot commented Jun 17, 2026

Uh oh!

seungrokj commented Jun 17, 2026

Uh oh!

chunfangamd left a comment

Choose a reason for hiding this comment

Uh oh!

seungrokj commented Jun 17, 2026

Uh oh!

seungrokj commented Jun 17, 2026

Uh oh!

Oseltamivir left a comment

Choose a reason for hiding this comment

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

seungrokj commented Jun 12, 2026 •

edited by cursor Bot

Loading