Skip to content

fix(rocm): run a decode batch's rows as one quantized product #2253

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
inureyes merged 4 commits into from
Oct 8, 2026
Merged

fix(rocm): run a decode batch's rows as one quantized product #2253

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
3fd1654
feat(bench): print batch decode steps and occupancy per serving level
inureyes Oct 8, 2026
895592d
fix(rocm): run a decode batch's rows as one quantized product
inureyes Oct 8, 2026
474163e
docs(bench): publish the gfx1151 batched server decode investigation
inureyes Oct 8, 2026
742bcba
docs: add technical report for PR #2253
inureyes Oct 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
197 changes: 197 additions & 0 deletions TECHNICAL_REPORTS/2253-rocm-batched-decode-row-fold-20261009.en.md
View file
Open in desktop

Large diffs are not rendered by default.

197 changes: 197 additions & 0 deletions TECHNICAL_REPORTS/2253-rocm-batched-decode-row-fold-20261009.ko.md
View file
Open in desktop

Large diffs are not rendered by default.

3 changes: 3 additions & 0 deletions docs/benchmark_results/data/rocm-batched-decode-gfx1151-2026-10-08/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Raw data for rocm-batched-decode-gfx1151-2026-10-08.md

One directory per guarded session (`s1/` to `s6/`, described in the page's Environment section), each holding the harness, probe, model-level and `mlxcel-bench-decode` outputs and the server logs; `guard_s*.log` are the matching `scripts/rocm_gpu_guard.sh` logs (sessions with a rejected attempt show it, and only the clean attempt's outputs are kept). Kernel traces are not committed (hundreds of MB); the `*/trace*/pbd_kernel_stats.csv` files are rocprofv3's per-kernel summaries of the model-level runs. `harness/` has the session scripts (`run_all.sh` and `run_matrix.sh` for `s1/`, `run_s2.sh` to `run_s6.sh` for the others) and the trace analysis (`step_split.py` for the model-level decode window, `analyze_trace.py` with `profile_driver.py` for the server windows). Local paths are replaced by `<meas>` (the measurement directory) and the repository root.
1,082 changes: 1,082 additions & 0 deletions docs/benchmark_results/data/rocm-batched-decode-gfx1151-2026-10-08/guard_s1.log
View file
Open in desktop

Large diffs are not rendered by default.

145 changes: 145 additions & 0 deletions docs/benchmark_results/data/rocm-batched-decode-gfx1151-2026-10-08/guard_s2.log
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,145 @@
2026-10-08T22:08:53+0900 waiting for guard lock /tmp/mlxcel-rocm-gpu-guard.lock
2026-10-08T23:05:05+0900 acquired guard lock /tmp/mlxcel-rocm-gpu-guard.lock after 3372s
2026-10-08T23:05:05+0900 waiting for 60s of idle GPU and no compiler
2026-10-08T23:06:33+0900 idle for 60s: kfd proc empty and no compiler in every 1 Hz sample
2026-10-08T23:06:33+0900 attempt 1/6: start: <meas>/run_s2.sh <meas>/s2
2026-10-08T23:06:34+0900 sample 1: clean gpu=[]
2026-10-08T23:06:35+0900 sample 2: clean gpu=[2966547:qmm_batch_rows_]
2026-10-08T23:06:37+0900 sample 3: clean gpu=[2966547:qmm_batch_rows_]
2026-10-08T23:06:38+0900 sample 4: clean gpu=[2966547:qmm_batch_rows_]
2026-10-08T23:06:40+0900 sample 5: clean gpu=[2966547:qmm_batch_rows_]
2026-10-08T23:06:41+0900 sample 6: clean gpu=[2966547:qmm_batch_rows_]
2026-10-08T23:06:43+0900 sample 7: clean gpu=[2971221:qmm_batch_rows_]
2026-10-08T23:06:44+0900 sample 8: clean gpu=[2971221:qmm_batch_rows_]
2026-10-08T23:06:46+0900 sample 9: CONTENDED foreign_gpu=[] compilers=[2973995:cargo]
2026-10-08T23:06:47+0900 sample 10: clean gpu=[2971221:qmm_batch_rows_]
2026-10-08T23:06:49+0900 sample 11: clean gpu=[2975896:profile_batched]
2026-10-08T23:06:50+0900 sample 12: clean gpu=[2975896:profile_batched]
2026-10-08T23:06:52+0900 sample 13: clean gpu=[2975896:profile_batched]
2026-10-08T23:06:53+0900 sample 14: clean gpu=[2975896:profile_batched]
2026-10-08T23:06:55+0900 sample 15: clean gpu=[2975896:profile_batched]
2026-10-08T23:06:56+0900 sample 16: clean gpu=[2975896:profile_batched]
2026-10-08T23:06:58+0900 sample 17: clean gpu=[2975896:profile_batched]
2026-10-08T23:06:59+0900 sample 18: clean gpu=[2975896:profile_batched]
2026-10-08T23:07:01+0900 sample 19: clean gpu=[2975896:profile_batched]
2026-10-08T23:07:02+0900 sample 20: clean gpu=[2975896:profile_batched]
2026-10-08T23:07:04+0900 sample 21: clean gpu=[2975896:profile_batched]
2026-10-08T23:07:05+0900 sample 22: clean gpu=[2975896:profile_batched]
2026-10-08T23:07:07+0900 sample 23: clean gpu=[2975896:profile_batched]
2026-10-08T23:07:08+0900 sample 24: clean gpu=[2975896:profile_batched]
2026-10-08T23:07:10+0900 sample 25: clean gpu=[2975896:profile_batched]
2026-10-08T23:07:11+0900 sample 26: clean gpu=[2975896:profile_batched]
2026-10-08T23:07:13+0900 sample 27: clean gpu=[2975896:profile_batched]
2026-10-08T23:07:14+0900 sample 28: clean gpu=[2975896:profile_batched]
2026-10-08T23:07:16+0900 sample 29: clean gpu=[2975896:profile_batched]
2026-10-08T23:07:17+0900 sample 30: clean gpu=[2975896:profile_batched]
2026-10-08T23:07:19+0900 sample 31: clean gpu=[2975896:profile_batched]
2026-10-08T23:07:20+0900 sample 32: clean gpu=[2975896:profile_batched]
2026-10-08T23:07:22+0900 sample 33: clean gpu=[2975896:profile_batched]
2026-10-08T23:07:24+0900 sample 34: clean gpu=[2975896:profile_batched]
2026-10-08T23:07:25+0900 sample 35: clean gpu=[2975896:profile_batched]
2026-10-08T23:07:27+0900 sample 36: clean gpu=[2975896:profile_batched]
2026-10-08T23:07:28+0900 sample 37: clean gpu=[2975896:profile_batched]
2026-10-08T23:07:30+0900 sample 38: clean gpu=[2975896:profile_batched]
2026-10-08T23:07:31+0900 sample 39: clean gpu=[2975896:profile_batched]
2026-10-08T23:07:33+0900 sample 40: clean gpu=[2975896:profile_batched]
2026-10-08T23:07:34+0900 sample 41: clean gpu=[2975896:profile_batched]
2026-10-08T23:07:36+0900 sample 42: clean gpu=[2975896:profile_batched]
2026-10-08T23:07:37+0900 sample 43: clean gpu=[2975896:profile_batched]
2026-10-08T23:07:39+0900 sample 44: clean gpu=[2975896:profile_batched]
2026-10-08T23:07:40+0900 sample 45: clean gpu=[2975896:profile_batched]
2026-10-08T23:07:42+0900 sample 46: clean gpu=[2975896:profile_batched]
2026-10-08T23:07:43+0900 sample 47: clean gpu=[2975896:profile_batched]
2026-10-08T23:07:45+0900 sample 48: clean gpu=[2975896:profile_batched]
2026-10-08T23:07:46+0900 sample 49: clean gpu=[2975896:profile_batched]
2026-10-08T23:07:48+0900 sample 50: clean gpu=[2975896:profile_batched]
2026-10-08T23:07:49+0900 sample 51: clean gpu=[2975896:profile_batched]
2026-10-08T23:07:51+0900 sample 52: clean gpu=[2975896:profile_batched]
2026-10-08T23:07:52+0900 sample 53: clean gpu=[2975896:profile_batched]
2026-10-08T23:07:54+0900 sample 54: clean gpu=[2975896:profile_batched]
2026-10-08T23:07:56+0900 sample 55: clean gpu=[2975896:profile_batched]
2026-10-08T23:07:57+0900 sample 56: clean gpu=[2975896:profile_batched]
2026-10-08T23:07:59+0900 sample 57: clean gpu=[2975896:profile_batched]
2026-10-08T23:08:00+0900 sample 58: clean gpu=[2975896:profile_batched]
2026-10-08T23:08:02+0900 sample 59: clean gpu=[3020808:profile_batched]
2026-10-08T23:08:03+0900 sample 60: clean gpu=[3020808:profile_batched]
2026-10-08T23:08:05+0900 sample 61: clean gpu=[3020808:profile_batched]
2026-10-08T23:08:06+0900 sample 62: clean gpu=[3020808:profile_batched]
2026-10-08T23:08:08+0900 sample 63: clean gpu=[3020808:profile_batched]
2026-10-08T23:08:09+0900 sample 64: clean gpu=[3020808:profile_batched]
2026-10-08T23:08:11+0900 sample 65: clean gpu=[3020808:profile_batched]
2026-10-08T23:08:13+0900 sample 66: clean gpu=[3020808:profile_batched]
2026-10-08T23:08:14+0900 sample 67: clean gpu=[3020808:profile_batched]
2026-10-08T23:08:15+0900 monitor: 67 samples
2026-10-08T23:08:15+0900 attempt 1: REJECTED (contended), exit 0; rerunning
2026-10-08T23:08:15+0900 waiting for 60s of idle GPU and no compiler
2026-10-08T23:09:43+0900 idle for 60s: kfd proc empty and no compiler in every 1 Hz sample
2026-10-08T23:09:43+0900 attempt 2/6: start: <meas>/run_s2.sh <meas>/s2
2026-10-08T23:09:44+0900 sample 1: clean gpu=[]
2026-10-08T23:09:45+0900 sample 2: clean gpu=[3084460:qmm_batch_rows_]
2026-10-08T23:09:47+0900 sample 3: clean gpu=[3084460:qmm_batch_rows_]
2026-10-08T23:09:48+0900 sample 4: clean gpu=[3084460:qmm_batch_rows_]
2026-10-08T23:09:50+0900 sample 5: clean gpu=[3084460:qmm_batch_rows_]
2026-10-08T23:09:51+0900 sample 6: clean gpu=[3089150:qmm_batch_rows_]
2026-10-08T23:09:53+0900 sample 7: clean gpu=[3089150:qmm_batch_rows_]
2026-10-08T23:09:54+0900 sample 8: clean gpu=[3089150:qmm_batch_rows_]
2026-10-08T23:09:56+0900 sample 9: clean gpu=[3089150:qmm_batch_rows_]
2026-10-08T23:09:57+0900 sample 10: clean gpu=[3089150:qmm_batch_rows_]
2026-10-08T23:09:59+0900 sample 11: clean gpu=[3093872:profile_batched]
2026-10-08T23:10:00+0900 sample 12: clean gpu=[3093872:profile_batched]
2026-10-08T23:10:02+0900 sample 13: clean gpu=[3093872:profile_batched]
2026-10-08T23:10:03+0900 sample 14: clean gpu=[3093872:profile_batched]
2026-10-08T23:10:05+0900 sample 15: clean gpu=[3093872:profile_batched]
2026-10-08T23:10:06+0900 sample 16: clean gpu=[3093872:profile_batched]
2026-10-08T23:10:08+0900 sample 17: clean gpu=[3093872:profile_batched]
2026-10-08T23:10:09+0900 sample 18: clean gpu=[3093872:profile_batched]
2026-10-08T23:10:11+0900 sample 19: clean gpu=[3093872:profile_batched]
2026-10-08T23:10:12+0900 sample 20: clean gpu=[3093872:profile_batched]
2026-10-08T23:10:14+0900 sample 21: clean gpu=[3093872:profile_batched]
2026-10-08T23:10:15+0900 sample 22: clean gpu=[3093872:profile_batched]
2026-10-08T23:10:17+0900 sample 23: clean gpu=[3093872:profile_batched]
2026-10-08T23:10:18+0900 sample 24: clean gpu=[3093872:profile_batched]
2026-10-08T23:10:20+0900 sample 25: clean gpu=[3093872:profile_batched]
2026-10-08T23:10:21+0900 sample 26: clean gpu=[3093872:profile_batched]
2026-10-08T23:10:23+0900 sample 27: clean gpu=[3093872:profile_batched]
2026-10-08T23:10:24+0900 sample 28: clean gpu=[3093872:profile_batched]
2026-10-08T23:10:26+0900 sample 29: clean gpu=[3093872:profile_batched]
2026-10-08T23:10:27+0900 sample 30: clean gpu=[3093872:profile_batched]
2026-10-08T23:10:29+0900 sample 31: clean gpu=[3093872:profile_batched]
2026-10-08T23:10:30+0900 sample 32: clean gpu=[3093872:profile_batched]
2026-10-08T23:10:32+0900 sample 33: clean gpu=[3093872:profile_batched]
2026-10-08T23:10:34+0900 sample 34: clean gpu=[3093872:profile_batched]
2026-10-08T23:10:35+0900 sample 35: clean gpu=[3093872:profile_batched]
2026-10-08T23:10:37+0900 sample 36: clean gpu=[3093872:profile_batched]
2026-10-08T23:10:38+0900 sample 37: clean gpu=[3093872:profile_batched]
2026-10-08T23:10:40+0900 sample 38: clean gpu=[3093872:profile_batched]
2026-10-08T23:10:41+0900 sample 39: clean gpu=[3093872:profile_batched]
2026-10-08T23:10:43+0900 sample 40: clean gpu=[3093872:profile_batched]
2026-10-08T23:10:44+0900 sample 41: clean gpu=[3093872:profile_batched]
2026-10-08T23:10:46+0900 sample 42: clean gpu=[3093872:profile_batched]
2026-10-08T23:10:47+0900 sample 43: clean gpu=[3093872:profile_batched]
2026-10-08T23:10:49+0900 sample 44: clean gpu=[3093872:profile_batched]
2026-10-08T23:10:50+0900 sample 45: clean gpu=[3093872:profile_batched]
2026-10-08T23:10:52+0900 sample 46: clean gpu=[3093872:profile_batched]
2026-10-08T23:10:53+0900 sample 47: clean gpu=[3093872:profile_batched]
2026-10-08T23:10:55+0900 sample 48: clean gpu=[3093872:profile_batched]
2026-10-08T23:10:56+0900 sample 49: clean gpu=[3093872:profile_batched]
2026-10-08T23:10:58+0900 sample 50: clean gpu=[3093872:profile_batched]
2026-10-08T23:10:59+0900 sample 51: clean gpu=[3093872:profile_batched]
2026-10-08T23:11:01+0900 sample 52: clean gpu=[3093872:profile_batched]
2026-10-08T23:11:02+0900 sample 53: clean gpu=[3093872:profile_batched]
2026-10-08T23:11:04+0900 sample 54: clean gpu=[3093872:profile_batched]
2026-10-08T23:11:06+0900 sample 55: clean gpu=[3093872:profile_batched]
2026-10-08T23:11:07+0900 sample 56: clean gpu=[3093872:profile_batched]
2026-10-08T23:11:09+0900 sample 57: clean gpu=[3093872:profile_batched]
2026-10-08T23:11:10+0900 sample 58: clean gpu=[3138110:profile_batched]
2026-10-08T23:11:12+0900 sample 59: clean gpu=[3138110:profile_batched]
2026-10-08T23:11:13+0900 sample 60: clean gpu=[3138110:profile_batched]
2026-10-08T23:11:15+0900 sample 61: clean gpu=[3138110:profile_batched]
2026-10-08T23:11:17+0900 sample 62: clean gpu=[3138110:profile_batched]
2026-10-08T23:11:18+0900 sample 63: clean gpu=[3138110:profile_batched]
2026-10-08T23:11:20+0900 sample 64: clean gpu=[3138110:profile_batched]
2026-10-08T23:11:21+0900 sample 65: clean gpu=[3138110:profile_batched]
2026-10-08T23:11:23+0900 sample 66: clean gpu=[3138110:profile_batched]
2026-10-08T23:11:24+0900 monitor: 66 samples
2026-10-08T23:11:24+0900 attempt 2: CLEAN, exit 0
Loading
Loading