Skip to content

Eval bug: Running large models on Vega 8 iGPU with Vulkan errors vk::Queue::submit: ErrorDeviceLost after ~ 50K context size #26447

Closed as not planned
Closed as not planned
Eval bug: Running large models on Vega 8 iGPU with Vulkan errors vk::Queue::submit: ErrorDeviceLost after ~ 50K context size#26447
Labels
bug-unconfirmedstale

Description

@cjdell
cjdell
opened on Aug 2, 2026

Name and Version

Always crashes at around 50K tokens of context. With smaller models I can get to maybe 60K tokens despite using less than 50% of the available GTT.

OS: NixOS 26.05 / Linux 7.1.5

Llama.cpp Version SHA: 11924d4c17abc27383376a1ac6a24fa3e36c1c0c
Llama.cpp Args: -t 12 -ngl all --ctx-size 131072

amdgpu_top: GTT: 62835 / 90112 MB (just before crash)

Operating systems

Linux

GGML backends

Vulkan

Hardware

Ryzen 5700G with Vega 8 with 96GB RAM (88GB allocated to GTT)

Models

Laguna-S-2.1-UD-IQ4_NL

Qwen3.6-27B-Fable-Fus-711-UnHeretic-NM-DAU-NEO-MAX-NEO-MTP-Q4_K_M

Problem description & steps to reproduce

Run llama-server -m Laguna-S-2.1-UD-IQ4_NL-00001-of-00003.gguf -t 12 -ngl all --ctx-size 131072 and start a coding session (i.e. OpenCode) and push the context size to around 50K.

First Bad Commit

No response

Relevant log output

Logs 
474.25.864.648 I slot      release: id  2 | task 10300 | stop processing: n_tokens = 49189, truncated = 0
474.26.077.013 I slot get_availabl: id  2 | task -1 | selected slot by LCP similarity, f_sim_best = 0.983 (> 0.100 thold), f_keep = 1.000
474.26.077.872 I slot launch_slot_: id  2 | task 10448 | processing task, is_child = 0
474.52.024.598 I slot print_timing: id  2 | task 10448 | prompt processing, n_tokens =    346, progress = 0.99, t =  25.95 s / 13.34 tokens per second
475.28.987.914 I slot print_timing: id  2 | task 10448 | prompt processing, n_tokens =    858, progress = 1.00, t =  62.91 s / 13.64 tokens per second
475.53.670.040 I slot print_timing: id  2 | task 10448 | prompt eval time =   67365.54 ms /   862 tokens (   78.15 ms per token,    12.80 tokens per second)
475.53.670.044 I slot print_timing: id  2 | task 10448 |        eval time =   20226.26 ms /    97 tokens (  208.52 ms per token,     4.80 tokens per second)
475.53.670.044 I slot print_timing: id  2 | task 10448 |       total time =   87591.80 ms /   959 tokens
475.53.670.045 I slot print_timing: id  2 | task 10448 |    graphs reused =      10324
475.53.671.335 I slot      release: id  2 | task 10448 | stop processing: n_tokens = 50147, truncated = 0
475.53.893.620 I slot get_availabl: id  2 | task -1 | selected slot by LCP similarity, f_sim_best = 0.948 (> 0.100 thold), f_keep = 1.000
475.53.894.488 I slot launch_slot_: id  2 | task 10548 | processing task, is_child = 0
radv/amdgpu: The CS has been cancelled because the context is lost. This context is innocent.
475.58.981.886 E srv  update_slots: decode() failed: vk::Queue::submit: ErrorDeviceLost
475.58.981.893 E srv    send_error: task id = 10548, error: decode() failed: vk::Queue::submit: ErrorDeviceLost
475.58.981.898 I slot      release: id  2 | task 10548 | stop processing: n_tokens = 52195, truncated = 0
475.58.981.933 W srv          stop: cancel task, id_task = 10548

[500694.802035] amdgpu 0000:07:00.0: Dumping IP State
[500694.803089] amdgpu 0000:07:00.0: Dumping IP State Completed
[500694.803096] amdgpu 0000:07:00.0: [drm] AMDGPU device coredump file has been created
[500694.803097] amdgpu 0000:07:00.0: [drm] Check your /sys/class/drm/card1/device/devcoredump/data
[500694.803098] amdgpu 0000:07:00.0: ring comp_1.2.1 timeout, signaled seq=2496578, emitted seq=2496581
[500694.803101] amdgpu 0000:07:00.0:  Process llama-server pid 106794 thread llama-server pid 106794
[500694.803102] amdgpu 0000:07:00.0: Starting comp_1.2.1 ring reset
[500694.803170] amdgpu 0000:07:00.0: Ring comp_1.2.1 reset succeeded
[500694.803171] amdgpu 0000:07:00.0: [drm] device wedged, but no recovery needed

[dump.txt](https://github.com/user-attachments/files/30630284/dump.txt)

Activity

  1. cjdell
    added
    bug-unconfirmed
    on Aug 2, 2026

  2. Crandel commented on Aug 4, 2026

    @Crandel
    Crandel
    on Aug 4, 2026

    Same result for me on AMD Ryzen AI 9 HX 470 + 96 Gb ram. For Laguna, Qwen and Gemma models

  3. cjdell commented on Aug 4, 2026

    @cjdell
    cjdell
    on Aug 4, 2026
    Author

    @Crandel Is it same using the ROCm build? I tried using ROCm on my Vega but it's barely supported and models don't load half the time. But for you with much newer hardware it might be smoother than using Vulkan.

  4. Crandel commented on Aug 5, 2026

    @Crandel
    Crandel
    on Aug 5, 2026

    @Crandel Is it same using the ROCm build? I tried using ROCm on my Vega but it's barely supported and models don't load half the time. But for you with much newer hardware it might be smoother than using Vulkan.

    No, I use Vulkan only. Maybe I should try it again. Last year it wasn't usable for me

  5. cjdell commented on Aug 6, 2026

    @cjdell
    cjdell
    on Aug 6, 2026
    Author

    I've set the kernel param amdgpu.lockup_timeout=10000 which defaults to 2000. I've exceeded 50K context and now at 67K and counting.

    The theory is that as the context grows, the round trip time for certain Vulkan calls starts to exceed 2000ms and for me this was around the 50K context mark.

    @Crandel Hopefully this will be the same solution for you.

  6. cjdell commented on Aug 6, 2026

    @cjdell
    cjdell
    on Aug 6, 2026
    Author

    @0cc4m I'm tagging you because I can see you've been investigating similar issues with DeviceLost. This might be part of the puzzle.

  7. 0cc4m commented on Aug 6, 2026

    @0cc4m
    0cc4m
    on Aug 6, 2026
    Contributor

    It's not a puzzle, that's what happens. Known issue. I'm trying to make it less likely to happen, but on slow AMD GPUs a single large operation can trigger it and I can't split that any further.

  8. cjdell commented on Aug 6, 2026

    @cjdell
    cjdell
    on Aug 6, 2026
    Author

    @0cc4m Thank you for your work. Now I know what to search for I can see this is the case.

    Can I humbly suggest this be added to some docs somewhere as possible workaround? I only ask because it would have save me 2 days of investigation and it may save others time too.

  9. 0cc4m commented on Aug 20, 2026

    @0cc4m
    0cc4m
    on Aug 20, 2026
    Contributor

    Sure, if you find a good place for a concise note feel free to open a PR.

  10. engrtipusultan commented on Sep 8, 2026

    @engrtipusultan
    engrtipusultan
    on Sep 8, 2026

    I've set the kernel param amdgpu.lockup_timeout=10000 which defaults to 2000. I've exceeded 50K context and now at 67K and counting.

    The theory is that as the context grows, the round trip time for certain Vulkan calls starts to exceed 2000ms and for me this was around the 50K context mark.

    @Crandel Hopefully this will be the same solution for you.

    For me that does not solve problem. I have increased the lockup timeout and set GGML_VK_MAX_NODES_PER_SUBMIT=1. For my vega8 49170 ctx seems to be the limit.

    bash  ./llama-bench -m /home/tipu/AI/models/unsloth/LFM25/LFM2.5-2.6B-QAD-Q4_0.gguf -ngl 100 --ubatch-size 1088 --batch-size 2048 -lm mlock -fa 1 -d 48000-65536+2048 -r 3ggml_vulkan: Found 1 Vulkan devices:
    ggml_vulkan: 0 = AMD Radeon Graphics (RADV RENOIR) (radv) | uma: 1 | fp16: 1 | bf16: 0 | fp4: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none

    model size params backend threads n_ubatch fa lm test t/s
    lfm2 2.6B Q4_0 1.48 GiB 2.70 B Vulkan,BLAS 8 1088 1 mlock pp512 @ d48000 66.77 ± 0.22
    lfm2 2.6B Q4_0 1.48 GiB 2.70 B Vulkan,BLAS 8 1088 1 mlock tg128 @ d48000 16.47 ± 0.01
    radv/amdgpu: The CS has been cancelled because the context is lost. This context is innocent.
    terminate called after throwing an instance of 'vk::DeviceLostError'
    what(): vk::Queue::submit: ErrorDeviceLost
    Aborted (core dumped)

  11. 0cc4m commented on Sep 8, 2026

    @0cc4m
    0cc4m
    on Sep 8, 2026
    Contributor

    Try running with GGML_VK_SERIALIZE_SUBMISSIONS=1, that should show more information about where it failed. Possibly your timeout is still too low, or there is some operator especially slow on your hardware.

  12. engrtipusultan commented on Sep 8, 2026

    @engrtipusultan
    engrtipusultan
    on Sep 8, 2026

    Thank you for looking into it.

    bash  GGML_VK_SERIALIZE_SUBMISSIONS=1 ./llama-bench -m /home/tipu/AI/models/unsloth/LFM25/LFM2.5-2.6B-QAD-Q4_0.gguf -ngl 100 -lm mlock -fa 1 --ubatch-size 1088 --batch-size 2048 -d 65536 -r 3
    ggml_vulkan: Found 1 Vulkan devices:
    ggml_vulkan: 0 = AMD Radeon Graphics (RADV RENOIR) (radv) | uma: 1 | fp16: 1 | bf16: 0 | fp4: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none

    model size params backend threads n_ubatch fa lm test t/s

    radv/amdgpu: The CS has been cancelled because the context is lost. This context is guilty of a hard recovery.
    terminate called after throwing an instance of 'vk::DeviceLostError'
    what(): vk::Queue::submit: ErrorDeviceLost
    Aborted (core dumped)

  13. engrtipusultan commented on Sep 8, 2026

    @engrtipusultan
    engrtipusultan
    on Sep 8, 2026

    Minor update. Reducing ub seems to help. I reduced ub to default 512. Now I can reach 65536 ctx.

    GGML_VK_SERIALIZE_SUBMISSIONS=1 ./llama-bench -m /home/tipu/AI/models/unsloth/LFM25/LFM2.5-2.6B-QAD-Q4_0.gguf -ngl 100 -lm mlock -fa 1 --ubatch-size 512 --batch-size 2048 -d 65536 -r 3

    ggml_vulkan: Found 1 Vulkan devices:
    ggml_vulkan: 0 = AMD Radeon Graphics (RADV RENOIR) (radv) | uma: 1 | fp16: 1 | bf16: 0 | fp4: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none

    model size params backend threads fa lm test t/s
    lfm2 2.6B Q4_0 1.48 GiB 2.70 B Vulkan,BLAS 8 1 mlock pp512 @ d65536 52.54 ± 1.15
    lfm2 2.6B Q4_0 1.48 GiB 2.70 B Vulkan,BLAS 8 1 mlock tg128 @ d65536 10.73 ± 2.25

    build: 5202104 (1)

    ./llama-bench -m /home/tipu/AI/models/unsloth/LFM25/LFM2.5-2.6B-QAD-Q4_0.gguf -ngl 100 -lm mlock -fa 1 --ubatch-size 512 --batch-size 2048 -d 65536 -r 3

    ggml_vulkan: Found 1 Vulkan devices:
    ggml_vulkan: 0 = AMD Radeon Graphics (RADV RENOIR) (radv) | uma: 1 | fp16: 1 | bf16: 0 | fp4: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none

    model size params backend threads fa lm test t/s
    lfm2 2.6B Q4_0 1.48 GiB 2.70 B Vulkan,BLAS 8 1 mlock pp512 @ d65536 51.73 ± 0.33
    lfm2 2.6B Q4_0 1.48 GiB 2.70 B Vulkan,BLAS 8 1 mlock tg128 @ d65536 14.52 ± 0.18

    build: 64e9bce (1)

  14. github-actions
    added
    stale
    on Oct 9, 2026

  15. github-actions commented on Oct 24, 2026

    @github-actions
    github-actionsBot
    on Oct 24, 2026 – with GitHub Actions
    Contributor

    This issue was closed because it has been inactive for 14 days since being marked as stale.

  16. github-actions
    closed this as not plannedon Oct 24, 2026

  17. elgueouatrijamal-hue commented on Oct 27, 2026

    @elgueouatrijamal-hue
    elgueouatrijamal-hue
    on Oct 27, 2026

    Thanks for sharing your Vega 8 / Vulkan troubleshooting report. It helped point me in the right direction for experimenting with local inference on AMD integrated graphics.

    My setup is more modest: Fedora 44, Ryzen 3 3200G, Radeon Vega 8, 13 GiB RAM, and llama.cpp built with Vulkan. I successfully ran Qwen2.5-3B-Instruct Q4_K_M, with all 37 model layers offloaded to the GPU.

    I haven't reproduced your large-context crash, as I'm currently using a 2048-token context. I wanted to share this successful small-model configuration in case it helps other users experimenting with older AMD APUs.

    Thanks again for documenting your findings!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bug-unconfirmedstale

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions