Repository navigation
CUDA/HIP stream usage on ROCm causes constant 100% GPU load - support disabling streams on ROCm #3929
Description
Prerequisites
Please answer the following questions for yourself before submitting an issue.
- I am running the latest code. Development is very rapid so there are no tagged versions as of now.
- I carefully followed the README.md.
- I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
- I reviewed the Discussions, and have a new bug or useful enhancement to share.
Feature Description
The CUDA/ROCm implementation in llama.cpp uses CUDA (HIP) streams for multi-GPU support - by default, 8 per GPU. It's fairly easy in the code to reduce this to 1, but there's no provision for disabling stream usage altogether. Unfortunately on ROCm, at least with RDNA 3 GPUs, streams appear to be buggy (ROCm/legacy-rocm-build#2625). It would be useful if stream use could be disabled, even at the cost of multi-GPU support also being disabled in this case.
Motivation
When streams are used on an AMD RDNA3 GPU with ROCm, as soon as either a 2nd stream is created, or any memory write takes place, the GPU is locked in a persistent 100% state with high power consumption, even when there's no computation happening. This is easily reproducible e.g. by loading a model using the llama.cpp server, and then never calling it, just letting it idle - the GPU draws power and heats up, as if under maximal load.
This is certainly a ROCm / amdgpu bug (reproducible using really simple minimal examples - see linked ROCm issue), but it would be nice if as a workaround, a "no-streams" mode would be available in llama.cpp.
Possible Implementation
As far as my limited understanding goes, CUDA/HIP Streams are essentially a GPU equivalent of threads. Based on ROCm example code, it's possible to do computation "in the main thread", i.e. without using streams, and this doesn't trigger the 100% GPU bug.
Activity
Googulator commentedon Nov 10, 2023
Found a workaround: by loading the amdgpu kernel module with the option "sched_policy=2", the power draw stays low both in llama.cpp and the attached examples. Unfortunately, this causes a slight increase of true idle power draw, i.e. when no ROCm program is running at all.
modinfo describes sched_policy as follows:
which has me a bit worried about deploying this workaround in production.
KerfuffleV2 commentedon Nov 10, 2023
kernel.org's description is just:
Set scheduling policy. Default is HWS(hardware scheduling) with over-subscription. Setting 1 disables over-subscription. Setting 2 disables HWS and statically assigns queues to HQDs. — https://www.kernel.org/doc/html/v6.6/gpu/amdgpu/module-parameters.html
I don't know if that means there's nothing to worry about, but the description doesn't really sound scary. Did you also try setting it to
1?Googulator commentedon Nov 10, 2023
Tried setting it to 1 - it doesn't change anything.
KerfuffleV2 commentedon Nov 10, 2023
Interesting. I found this: https://dl.acm.org/doi/fullHtml/10.1145/3453417.3453432
I only skimmed it, but the impression I get is that the scheduling stuff really only matters when there's contention for the GPU resources. Say for example a server where multiple users are sharing the GPU.
Googulator commentedon Nov 10, 2023
Unfortunately there is a rather unpleasant side effect: it makes the local console/framebuffer lag badly. Luckily our AMD machines are mostly remotely managed, but it could be really annoying for anyone accessing their consoles locally.
(We didn't test if there are any issues with X.)
65a commentedon Jan 1, 2024
I found setting
GPU_MAX_HW_QUEUES=1in the environment or commandline appears to workaround the issue with no impact on tok/s (didn't do a detailed analysis however). I would also like to see this fixed properly upstream though. I filed a bug at freedesktop foramdgpuas well.EDIT: A patch like https://pastebin.com/ug0R3nGP works to fix this issue locally here.
github-actions commentedon Apr 2, 2024
This issue was closed because it has been inactive for 14 days since being marked as stale.
common : default GPU_MAX_HW_QUEUES=1 on HIP to fix idle GPU load