Eval bug: vulkan: regression: vram usage increased

[rhjdvsgsgks]

Name and Version

716bd6d
bisected

Operating systems

Linux

GGML backends

Vulkan

Hardware

amdgpu 8g

Models

Qwen2.5-Coder-14B-Instruct-Q4_K_M
or any model with similar size

Problem description & steps to reproduce

on c250ecb . the weight part of model can fit into vram. left only context/kv cache on gtt. memory usage is 8166m vram + 2271m gtt.

but on 716bd6d . memory usage is 6342m vram + 4107m gtt. significantly slowed down the tg speed.

First Bad Commit

716bd6d

Relevant log output

no difference on log output

[jeffbolznv]

Sounds like the additional shaders pushed it just over the edge and some allocation no longer fits in vidmem. Maybe we should look into compiling shaders more on demand? Curious what @0cc4m thinks about that.

[0cc4m]

I was under the impression that shaders take up negligible space in VRAM. I guess maybe it pushed this case over the limit where the driver decides to move pages to RAM.

With the growing number of shaders it probably makes sense to look into on-demand loading pipelines, yeah.

Edit: Maybe we can gather the required shaders in the graph dry-run stage and compile those that haven't yet been compiled in a multi-threaded way before executing the graph?

[jeffbolznv]

They're each small, but we've grown to have quite a few variants. With the coopmat2 path I see about 800 total shader compiles taking around 56MB on my system. The new shaders in 716bd6d are about 12MB of that.

Edit: Maybe we can gather the required shaders in the graph dry-run stage and compile those that haven't yet been compiled in a multi-threaded way before executing the graph?

That's what I was thinking, my concern is just whether this violates the "Zero memory allocations during runtime" feature of ggml. But I guess memory allocation also happens as part of the dryrun so it should be OK?

[0cc4m]

Edit: Maybe we can gather the required shaders in the graph dry-run stage and compile those that haven't yet been compiled in a multi-threaded way before executing the graph?

That's what I was thinking, my concern is just whether this violates the "Zero memory allocations during runtime" feature of ggml. But I guess memory allocation also happens as part of the dryrun so it should be OK?

The temporary buffers also get allocated there, so we're not following that either way. It would be fine.

[ggerganov]

The "zero memory allocations during runtime" refers only to the ggml-base library, without the backends. So it is OK.

[rhjdvsgsgks]

i can confirm it solved my problem. thank you