Skip to content

[ICS] Eval bug: OpenVINO: low tg #308

Open
Open
[ICS] Eval bug: OpenVINO: low tg#308

Description

@haarika-madaka
haarika-madaka
opened on Sep 3, 2026

Name and Version

root@ae0463242527:/app# ./llama-cli --version
version: 0.2.0-dev (build 10615, commit f280b2698)
built with GNU 13.3.0 for Linux x86_64

Operating systems

Linux

GGML backends

OpenVINO

Hardware

B580 + 7 5700X3D + 48GB at 3133MHz

Models

Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf, Qwen3-14B-Q4_K_M.gguf, Qwen3.5-0.8B-Q4_K_M.gguf, Qwen3.5-9B-Q4_K_M.gguf

Problem description & steps to reproduce

The tg on llama.cpp OpenVINO is lower than ovms and OpenArc

First Bad Commit

No response

Relevant log output

llama.cpp 
root@ae0463242527:/app# GGML_OPENVINO_DEVICE=GPU ./llama-bench --model /models/Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf 
OpenVINO: using device GPU
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| qwen2 14B Q4_K - Medium        |   8.37 GiB |    14.77 B | OPENVINO   |  -1 |           pp512 |      2491.79 ± 43.30 |
| qwen2 14B Q4_K - Medium        |   8.37 GiB |    14.77 B | OPENVINO   |  -1 |           tg128 |         22.91 ± 0.39 |

build: f280b2698 (10615)
root@ae0463242527:/app# GGML_OPENVINO_DEVICE=GPU ./llama-bench --model /models/Qwen3-14B-Q4_K_M.gguf 
OpenVINO: using device GPU
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| qwen3 14B Q4_K - Medium        |   8.38 GiB |    14.77 B | OPENVINO   |  -1 |           pp512 |      2546.33 ± 46.69 |
| qwen3 14B Q4_K - Medium        |   8.38 GiB |    14.77 B | OPENVINO   |  -1 |           tg128 |         24.63 ± 0.47 |

build: f280b2698 (10615)
root@ae0463242527:/app# GGML_OPENVINO_DEVICE=GPU ./llama-bench --model /models/Qwen3.5-0.8B-Q4_K_M.gguf 
OpenVINO: using device GPU
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| qwen35 0.8B Q4_K - Medium      | 497.39 MiB |   752.39 M | OPENVINO   |  -1 |           pp512 |     8825.12 ± 138.33 |
| qwen35 0.8B Q4_K - Medium      | 497.39 MiB |   752.39 M | OPENVINO   |  -1 |           tg128 |         36.80 ± 0.22 |

build: f280b2698 (10615)
root@ae0463242527:/app# GGML_OPENVINO_DEVICE=GPU ./llama-bench --model /models/Qwen3.5-9B-Q4_K_M.gguf   
OpenVINO: using device GPU
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| qwen35 9B Q4_K - Medium        |   5.28 GiB |     8.95 B | OPENVINO   |  -1 |           pp512 |      2996.08 ± 17.29 |
| qwen35 9B Q4_K - Medium        |   5.28 GiB |     8.95 B | OPENVINO   |  -1 |           tg128 |         18.40 ± 0.67 |

build: f280b2698 (10615)
ovms

podman run --interactive --tty --publish=1235:8080 --rm --device=/dev/dri/renderD128 --user root docker.io/openvino/model_server:2026.3-gpu --rest_port 8080 --model_repository_path /models/ --source_model <model> --target_device GPU --task text_generation

podman run --interactive --tty --rm localhost/llama-benchy:latest --base-url http://host.docker.internal:1235/v3 --pp 512 --tg 128 --runs 5

| model                                       |   test |              t/s |     peak t/s |      ttfr (ms) |   est_ppt (ms) |   e2e_ttft (ms) |
|:--------------------------------------------|-------:|-----------------:|-------------:|---------------:|---------------:|----------------:|
| OpenVINO/Qwen2.5-Coder-14B-Instruct-int4-ov |  pp512 | 2486.48 ± 215.33 |              | 210.07 ± 18.32 | 207.50 ± 18.32 |  223.74 ± 18.92 |
| OpenVINO/Qwen2.5-Coder-14B-Instruct-int4-ov |  tg128 |     47.03 ± 5.39 | 48.49 ± 7.51 |                |                |                 |
| model                      |   test |            t/s |     peak t/s |     ttfr (ms) |   est_ppt (ms) |   e2e_ttft (ms) |
|:---------------------------|-------:|---------------:|-------------:|--------------:|---------------:|----------------:|
| OpenVINO/Qwen3-14B-int4-ov |  pp512 | 2690.01 ± 6.41 |              | 191.19 ± 0.45 |  190.33 ± 0.45 |   197.48 ± 4.81 |
| OpenVINO/Qwen3-14B-int4-ov |  tg128 |   41.63 ± 0.12 | 42.00 ± 0.00 |               |                |                 |
| model                         |   test |              t/s |      peak t/s |      ttfr (ms) |   est_ppt (ms) |   e2e_ttft (ms) |
|:------------------------------|-------:|-----------------:|--------------:|---------------:|---------------:|----------------:|
| OpenVINO/Qwen3.5-0.8B-int4-ov |  pp512 | 3613.92 ± 487.60 |               | 144.97 ± 18.60 | 144.35 ± 18.60 |  144.97 ± 18.60 |
| OpenVINO/Qwen3.5-0.8B-int4-ov |  tg128 |    133.51 ± 1.79 | 134.56 ± 1.80 |                |                |                 |
| model                       |   test |              t/s |     peak t/s |      ttfr (ms) |   est_ppt (ms) |   e2e_ttft (ms) |
|:----------------------------|-------:|-----------------:|-------------:|---------------:|---------------:|----------------:|
| OpenVINO/Qwen3.5-9B-int4-ov |  pp512 | 2098.39 ± 253.55 |              | 249.45 ± 27.33 | 247.76 ± 27.33 |  256.70 ± 20.08 |
| OpenVINO/Qwen3.5-9B-int4-ov |  tg128 |     61.06 ± 0.34 | 61.80 ± 0.40 |                |                |                 |

Activity

  1. github-project-automation
    added this to LlamaCPP_OpenVINO Backend Projecton Sep 3, 2026
  2. github-project-automation
    moved this to Backlog in LlamaCPP_OpenVINO Backend Projecton Sep 3, 2026

  3. haarika-madaka commented on Sep 7, 2026

    @haarika-madaka
    haarika-madaka
    on Sep 7, 2026
    Author

    Need to replace one-by-one:

    OpenVINO/Qwen2.5-Coder-14B-Instruct-int4-ov
    OpenVINO/Qwen3-14B-int4-ov
    OpenVINO/Qwen3.5-0.8B-int4-ov
    OpenVINO/Qwen3.5-9B-int4-ov

    First terminal:
    podman run --interactive --tty --publish=1235:8080 --rm --device=/dev/dri --user root [docker.io/openvino/model_server:2026.3-gpu](https://docker.io/openvino/model_server:2026.3-gpu) --rest_port 8080 --model_repository_path /models/ --source_model <model> --target_device GPU --task text_generation

    Second terminal:
    llama-benchy --base-url http://localhost:1235/v3 --pp 512 --tg 128 --runs 5

    We ran all the tests (OVMS and llama.cpp) on Lunar Lake machine. Even though the GPU device is different from the customer (B580), but the tg obtained from OVMS and llama.cpp is obvious.

    llama.cpp
    Image

    OVMS
    Image

    Things to highlight:

    1. Customer using llama-benchy (https://github.com/eugr/llama-benchy) for the OVMS test
    2. OpenVINO IR is used in OVMS test while GGUF model is used in llama.cpp test but the models should be the same just different in format
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions