ci : add self-hosted webgpu to hf-jobs

[CISC]

Overview

cont #28258

Additional information

Turns out it was simpler than initially thought, Docker Vulkan passthrough works OOTB, thanks to @olliewalsh for pointing this out.

Disabled CM hf-jobs for now.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: Təwə
• Language disclosure: Malay (Rejang script)

[netrunnereve]

To be fair I've seen it timeout on our selfhosted t4 as well in the past https://github.com/ggml-org/llama.cpp/actions/runs/34319079872/job/102361526168, I don't know if it's just slow or if certain tests get stuck.

[0cc4m]

That may just be shader caching. The nvidia driver is not fast with compiles, a run can be significantly slower if it loads all or many for the first time.

[ggerganov]

Yeah, the lack of Vulkan shader cache makes it slow. The ephemeral HF jobs runners don't have a persistent storage like the self-hosted T4 runners, so every new run will start with a cold cache.

Hopefully the test-backend-ops -j N option would result in a significant improvement in this use case - IIUC the shader compilation is CPU-bound.

[CISC]

Yeah, the lack of Vulkan shader cache makes it slow. The ephemeral HF jobs runners don't have a persistent storage like the self-hosted T4 runners, so every new run will start with a cold cache.

Ah, is this cache stored somewhere trivial so we can put it in buckets? Either way t4-medium should help a lot when the -j option lands.

[ggerganov]

It should be stored in $HOME/.cache/nvidia/GLCache/ per Jeff's comment. I think depending on the outcome of enabling -j, we can consider caching or not the shaders.

I am preparing a PR that will extract the test-backend-ops runs into a separate CI workflow so we can control it more precisely.

[jeffbolznv]

I think it should be fine to use -j for vulkan testing, and several months ago I made it so the compiles happen outside of any locks, so it should help with compile time. Not sure about other backends.

There is an env var to override the shader cache location, but if there is no persistent storage I guess that won't help.

[CISC]

There is an env var to override the shader cache location, but if there is no persistent storage I guess that won't help.

The persistent storage issue may have been fixed, so it can probably be enabled if need be.

[ggerganov]

Let's merge #28740, then rebase this PR on master and see if the -j helped.

[netrunnereve]

I think it should be fine to use -j for vulkan testing, and several months ago I made it so the compiles happen outside of any locks, so it should help with compile time. Not sure about other backends.

With -j 4 a test-backend-ops on my 470 finishes in around 5 minutes versus the 11 minutes or so for a regular run. No issues so far.

[ggerganov]

Hm, looks like the gpu-vulkan-nvidia-cm job deadlocked now with -j 7: https://github.com/ggml-org/llama.cpp/actions/runs/34697303026/job/103562882003?pr=28712#step:6:24943

[ggerganov]

Hm, and another Vulkan job also deadlocked, even without -j: https://github.com/ggml-org/ggml/actions/runs/34689579337/job/103542363483?pr=1626

[CISC]

Hm, looks like the gpu-vulkan-nvidia-cm job deadlocked now with -j 7: https://github.com/ggml-org/llama.cpp/actions/runs/34697303026/job/103562882003?pr=28712#step:6:24943

Yep, got killed after 2 hour limit:

2026-09-12 13:47:56Z: Running job: gpu-vulkan-nvidia-cm
2026-09-12 15:58:46Z: Job gpu-vulkan-nvidia-cm completed with result: Canceled

[ggerganov]

Yeah, I suspect some weird regression in Vulkan. Here is a 3rd job that got stuck earlier today: https://github.com/ggml-org/llama.cpp/actions/runs/34686526464/job/103534353964

[jeffbolznv]

I'm trying to debug the original device lost reported in #28740. I haven't seen any failures where it just deadlocks. It can crash after the device lost, not sure how that would show up in CI.

[ggerganov]

IMO we can merge this. The runtimes are bit large (almost 1h), but the jobs don't utilize the ccache yet since we create is only on the master branch. So the runtime will be less after we merge.

Note that the HF runners are using the 580 NVIDIA driver. Not sure if there is a simple way to upgrade them to the new 615 driver that we recently installed on the old Azure runners.

Alternatively, we can split the work like this:

• Keep the existing Azure Tesla T4 runners (with the new 615 driver) running the gpu-vulkan-nvidia-cm and gpu-vulkan-nvidia-cm2 jobs
• Move only the gpu-webgpu-nvidia job to the HF runners

This way we lift some of the pressure from the Azure runners (by moving the webgpu jobs to HF) + keep running the Vulkan CI jobs on the new driver.

Eventually, I would like to retire the Azure runners since the sponsorship will end sometime in 2027. But we can still keep them running for a little while until that happens.

[CISC]

Think there's much point in running them on t4-medium, or should we go back to t4-small?

[ggerganov]

t4-small is probably OK. We can compare it against the last t4-medium run:

https://github.com/ggml-org/llama.cpp/actions/runs/34751007440?pr=28712

[CISC]

Ok, so definitely slower, but most of that is build time, so should not be too bad once cache is in play.

[CISC]

• Keep the existing Azure Tesla T4 runners (with the new 615 driver) running the gpu-vulkan-nvidia-cm and gpu-vulkan-nvidia-cm2 jobs
• Move only the gpu-webgpu-nvidia job to the HF runners

This way we lift some of the pressure from the Azure runners (by moving the webgpu jobs to HF) + keep running the Vulkan CI jobs on the new driver.

Eventually, I would like to retire the Azure runners since the sponsorship will end sometime in 2027. But we can still keep them running for a little while until that happens.

So, do this for now and comment out the new cm jobs for future use?

[ggerganov]

So, do this for now and comment out the new cm jobs for future use?

Yes, let's do it like this.

[CISC]

@ggerganov I think maybe Azure runners need a restart, seems to be stuck/slow, I just cancelled a bunch of queued jobs from yesterday.

[ggerganov]

Yeah, there was a job in the morning stuck for 4h. I am not sure if this is somehow related to the issue that @jeffbolznv has identified: ggml-org/ggml#1626 (comment).

To summarize, so far we observe:

• Intermittent ARGSORT failures
• Some jobs getting stuck (I think mainly CM2, rather than CM, though not 100% sure)
• Random crash in one of the whisper.cpp vulkan jobs

[CISC]

Just found this one stuck for 5hrs+
https://github.com/ggml-org/llama.cpp/actions/runs/34939723411/job/104285387342

[jeffbolznv]

Just found this one stuck for 5hrs+ https://github.com/ggml-org/llama.cpp/actions/runs/34939723411/job/104285387342

Is the assert failure not being handled by the CI scripts?

[CISC]

Just found this one stuck for 5hrs+ https://github.com/ggml-org/llama.cpp/actions/runs/34939723411/job/104285387342

Is the assert failure not being handled by the CI scripts?

Usually is, no idea why it got stuck.

[netrunnereve]

Usually is, no idea why it got stuck.

Isn't there some kind of timeout we can set? There's no reason why that job should run longer than an hour.

[CISC]

Usually is, no idea why it got stuck.

Isn't there some kind of timeout we can set? There's no reason why that job should run longer than an hour.

Apparently one can set timeout-minutes on a job, it defaults to 6hrs though so I guess it would have timed out on its own shortly afterwards if I hadn't cancelled it.