Skip to content

openvino: merge refactor PR - isolate op support, manage buffers, document API - #284

Draft
zhaixuejun1993 with Copilot wants to merge 135 commits into
dev_backend_openvinofrom
copilot/xuejunrefactor-ggml-openvino
Draft

zhaixuejun1993 with Copilot wants to merge 135 commits into
dev_backend_openvinofrom
copilot/xuejunrefactor-ggml-openvino

Conversation

Copilot AI commented Aug 14, 2026

Copy link
Copy Markdown

Ports and rebases PR #278 (zhaixuejun1993/llama.cpp:xuejun/refactor-ggml-openvino) onto the latest dev_backend_openvino, resolving all merge conflicts.

Overview

Structural refactor of the OpenVINO backend, splitting a monolithic ggml-openvino.cpp into focused modules, plus conflict resolution to incorporate fixes that landed on dev_backend_openvino after the PR diverged.

Changes

  • New modules extracted from ggml-openvino.cpp:

    • ggml-openvino-op-support.cpp/h - op support policy (ggml_openvino_device_supports_op_impl), previously inline; includes full per-op policy table in header comments
    • ggml-openvino-buffer.cpp/h - buffer allocation and tensor extra management
    • ggml-openvino-weight-buffer-release.cpp/h - GGML_OPENVINO_RELEASE_WEIGHTS RSS reclaim logic
  • ggml-openvino.cpp: now a thin orchestration layer; device_supports_op delegates to ggml_openvino_device_supports_op_impl; buffer_type_context gains bool is_host field

  • Conflict resolutions:

    • ggml-openvino-op-support.cpp (GGML_OP_CPY): added quantized-destination guard from ae02f5e after the existing BF16 src/dst check:
      // CPY to a quantized destination (e.g. f32 -> q4_0) is numerically unstable with OpenVINO backend.
      if (ggml_is_quantized(op->type)) {
          return true;
      }
    • utils.cpp (is_model_splitted): retained GGML_OPENVINO_ENABLE_FALLBACK env-var gate from c66a9c9
    • openvino/op/sqr.cpp: trailing newline preserved (matches base)
    • tests/test-llama-archs.cpp: auto-merged correctly; has_openvino skip removed, Meta tensor-split config always appended; new arch entries (QWEN3TTS, MUSE_GLIMMER, GRANITE_SWITCH) and updated WebGPU/HIP skip logic retained from base

Additional information

Merge commit on branch copilot/xuejunrefactor-ggml-openvino (rebased onto ravi9/llama.cpp:dev_backend_openvino at 37b164f).

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: YES - AI agent performed the merge, conflict resolution, and authored this description. The contributor is responsible for reviewing all changes before submission.

zhaixuejun1993 and others added 30 commits July 27, 2026 12:32
…g_src to recorde the src ggml tensor for OpenVINO dynamic shape infer
enable qwen35

Fix after rebase

remove logging
…t reason: the backend test initializes unary op inputs over a wide range, [-150, 150]. For FP32, exp(x) overflows around x ~= 88.7, so this test can randomly generate values right in or beyond the overflow region
In stateful mode the NEOX RoPE branch fed rank-3 data ([S, n_heads,
head_size]) into the Multiply against the rank-4 cos/sin tables
([1, S, 1, n_dims/2]). That mixed-rank broadcast is miscomputed by the
OpenVINO GPU plugin, corrupting the rotated Q/K and producing garbage
output (e.g. Phi-3-mini). Lift the data to rank-4 before the split/
Multiply so the operands are equal-rank, matching what the TYPE_NORMAL
branch already does. CPU and stateless paths are unaffected.

Phi-3-mini-Q4_K_M, wiki.test perplexity, GPU stateful:
  before: PPL = 27120.43
  after:  PPL = 6.2263   (CPU reference: 6.2251)
…ov name in ov bk; 3) fix issue in arch test & op test with latest code update
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
zhaixuejun1993 and others added 10 commits August 12, 2026 15:59
Add runtime configuration entries for the newly recognized OpenVINO environment variables.

Document GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR as the frontend compiled-model cache used to export and import compiled blobs for matching single-graph models.

Document GGML_OPENVINO_MEMORY_OPTIMIZE as the umbrella switch, including how GGML_OPENVINO_REDUCE_COMPILE_MEM and the GPU-only GGML_OPENVINO_RELEASE_WEIGHTS override or inherit from it.
1. GGML_OP_PAD was missing from compute_node_dynamic_dims(), causing a crash
on decode for models that pad the token embedding (n_embd -> n_embd_inp).
PAD never reorders/merges dims, so it keeps the same dynamic dim index as
its source.

2. process_view_input_new() chained VIEW inputs through src[0] (the
immediate op-graph parent) using offsets treated as relative to that
parent. But ggml_tensor::view_offs is always absolute from the true root
allocation (ggml collapses VIEW-of-VIEW chains internally). For the
per-layer deepstack view ("embd (view)", whose src[0] is "embd" - itself
an already-narrowed, zero-offset VIEW of the padded root, with the SAME
ggml shape as the deepstack view but a different absolute offset), this
caused an out-of-bounds re-slice that silently fell back to returning the
wrong (already-resolved sibling) tensor. In practice every deepstack ADD
ended up adding the real base token embedding into the residual stream
instead of zero, corrupting generation ("Hello my name is 1000000..."
instead of coherent text). Fixed by detecting this pattern (same shape as
the immediate src, different absolute offset) and re-slicing directly
from the untouched root tensor using the innermost view's absolute
offset.

Also adds a GGML_OPENVINO_DEBUG_NODE=<name1>,<name2>,... env var that attaches
extra debug Result nodes for arbitrary intermediate tensors, without binding
them to any ggml buffer (avoiding the risk of reading a ggml buffer that has
since been overwritten by a later in-place op). This was instrumental in
diagnosing bug #2 above and is left in as a general-purpose debugging aid.
IMROPE's inp_pos tensor packs 4 stacked t/h/w/e position planes into
ne[0] = 4*n_tokens instead of one value per token. On NPU's static-shape
path, inp_pos was padded/shaped as if it held a single plane, which
interleaved padding across the 4 planes and desynced later reshapes
from the rest of the (chunk_size-wide) graph.

- add GgmlOvDecoder::get_inp_pos_n_planes() to detect IMROPE's 4-plane layout
- get_graph_input_shape(): size inp_pos as n_planes * chunk_size (prefill)
  or n_planes (decode) instead of assuming 1 value per token
- get_ov_input_tensor_static_prefill(): pad each plane to chunk_size
  independently instead of one flat block
- get_ov_input_tensor_static_decode(): copy n_planes contiguous values
  instead of asserting/copying a single scalar
reject CPY with quantized destination as unsupported
…ent API

Assisted-by: Claude Sonnet

Co-authored-by: zhaixuejun1993 <52686861+zhaixuejun1993@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants