Optimize NPU performance with KV cache slicing and requantization

ci/run.sh

@@ -669,6 +669,11 @@ function gg_run_test_backend_ops {
         args_extra=""
     fi
 
+    # TODO: OpenVINO GPU plugin crashes (CL_OUT_OF_RESOURCES) with 2 concurrent workers on GPU.
+    if [ ! -z "${GG_BUILD_OPENVINO}" ] && [ "${GGML_OPENVINO_DEVICE:-}" = "GPU" ]; then
+        args_extra=""
+    fi
+
     # TODO: reduce the test-backend-ops timeout to 1800s
     if [ ! -z ${GG_BUILD_HIGH_PERF} ]; then
         (time timeout 3600 ./bin/test-backend-ops ${args_extra} -b CPU) 2>&1 | tee -a $OUT/${ci}-test-backend-ops.log

docs/backend/OPENVINO.md

@@ -715,14 +715,30 @@ Boolean flags follow a uniform convention: set to a **positive integer** (e.g. `
 | `GGML_OPENVINO_CACHE_DIR`         | String    | `not set`  | Directory for OpenVINO model caching (recommended: `/tmp/ov_cache`). Enables model caching when set. **Not supported on NPU devices.** |
 | `GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR` | String | `not set` | Directory for the frontend compiled-model cache. When set, OpenVINO compiled models are exported as blobs and imported on later runs to skip weight requantization, graph conversion, and compilation for matching single-graph models. |
 | `GGML_OPENVINO_PREFILL_CHUNK_SIZE`| Integer   | `256`      | Token chunk size for **NPU** prefill (NPU-only; ignored on CPU/GPU). Must be a positive integer; otherwise the default is used. |
-| `GGML_OPENVINO_NPU_COMPILE_CONFIG` | String | `not set` | NPU-only compiler mode parameters forwarded to OpenVINO as `NPU_COMPILATION_MODE_PARAMS`, for example `optimization-level=3`. |
+| `GGML_OPENVINO_NPU_CONFIG`       | String    | `not set`  | Recommended NPU plugin/compiler override entry point. Comma-separated `KEY=VALUE` pairs are applied after named NPU options, so they can override defaults or aliases; for example, `NPU_TILES=6,NPUW_SLICE_OUT=YES,NPU_COMPILATION_MODE_PARAMS=optimization-level=3`. |
+| `GGML_OPENVINO_NPU_COMPILE_CONFIG` | String | `not set` | Legacy alias for `GGML_OPENVINO_NPU_CONFIG=NPU_COMPILATION_MODE_PARAMS=...`, for example `optimization-level=3`. `GGML_OPENVINO_NPU_CONFIG` is preferred and overrides this alias when it sets the same property. |
+| `GGML_OPENVINO_NPU_COMPILER_TYPE` | String | `DRIVER` | Named alias for the `NPU_COMPILER_TYPE` property. Values include `PLUGIN`, `DRIVER`, and `PREFER_PLUGIN`; `GGML_OPENVINO_NPU_CONFIG` can override it. |
+| `GGML_OPENVINO_NPUW_FUNCALL_FOR_ALL` | String | OpenVINO default | Named alias for the `NPUW_FUNCALL_FOR_ALL` property. Useful for disabling funcall-for-all on driver/compiler combinations where it hurts stability or long-context performance; `GGML_OPENVINO_NPU_CONFIG` can override it. |
+| `GGML_OPENVINO_NPUW_UNFOLD_IREQS` | String | OpenVINO default | Named alias for the `NPUW_UNFOLD_IREQS` property. Unfolding can reduce repeated funcall dispatch overhead at the cost of higher memory use; `GGML_OPENVINO_NPU_CONFIG` can override it. |
+| `GGML_OPENVINO_COMPILATION_NUM_THREADS` | String | OpenVINO default | Named alias for the `COMPILATION_NUM_THREADS` property. Caps compiler worker parallelism to reduce peak host memory during large NPU graph compilation; `GGML_OPENVINO_NPU_CONFIG` can override it. |
+| `GGML_OPENVINO_NPU_REQUANT_POLICY`| String    | `group-128` | NPU-only weight requantization layout. Use `group-128` for `Q4_0_128` or experimental `channel-wise` for `Q4_0_C`. Any other value fails during model loading. |
+| `GGML_OPENVINO_NPU_FAST_MASK`     | Boolean   | `0`        | NPU-only. Build the static prefill attention mask directly into the input tensor, skipping an intermediate host allocation + full-tensor copy per chunk. Numerically identical to the default path. |
+| `GGML_OPENVINO_NPU_L0_HOST_TENSORS`| Boolean  | `0`        | NPU-only. Back the KV-cache / compute buffers with Level Zero host memory so the plugin imports them without an extra host-to-device copy each infer, cutting prefill KV I/O time. Falls back to plain host memory if the L0 allocation is unavailable. |
+| `GGML_OPENVINO_NPU_KV_SLICE`      | Boolean   | `0`        | NPU-only. Compile static prefill graphs and bind prefill KV-cache tensors at the active attention length instead of the full context capacity. Decode retains its fixed full-context graph. The prefill model is recompiled when the aligned attention length changes. |
 | `GGML_OPENVINO_STATEFUL_EXECUTION`| Boolean   | `0`        | Enable stateful KV cache for better performance. Recommended on CPU, GPU.                                   |
 | `GGML_OPENVINO_DISABLE_CACHE`     | Boolean   | `0`        | Disable the in-process compiled-model / decoder cache (cache is on by default). Set to `1` to disable.      |
 | `GGML_OPENVINO_DISABLE_KV_SLICE`  | Boolean   | `0`        | Disable the KV-cache input-tensor slicing optimization (slicing is on by default on CPU/GPU). Set to `1` to disable. |
+| `GGML_OPENVINO_KV_SCATTER_ELEMENTS` | Boolean | `0`        | Use `ScatterElementsUpdate` instead of `ScatterUpdate` for non-stateful single-row KV writes. Intended for NPU decode paths where `ScatterUpdate` copies the full destination tensor to update one row. |
+| `GGML_OPENVINO_DISABLE_KV_STATE_RELAYOUT` | Boolean | `0`     | Disable the stateful KV-state sequence-axis relayout (relayout is on by default). It moves the KV state sequence axis from dim 1 to dim 2, so the GPU plugin can append new tokens in place instead of copying the whole state every token, and the reader side no longer transposes the whole accumulated state. Set to `1` to disable. |
+| `GGML_OPENVINO_TOKEN_EMBD_I8`     | Boolean   | `0`        | NPU-only. Keep `token_embd.weight` from widening Q6_K to FP16 by using the normal int8 compressed-weight layout instead. Reduces embedding-table memory at possible accuracy/performance cost. |
+| `GGML_OPENVINO_TOKEN_EMBD_I4`     | Boolean   | `0`        | NPU-only. Requantize `token_embd.weight` to group-128 int4. Intended mainly for tied embeddings, where compressed-row gather can read only the selected packed rows before dequantization. |
+| `GGML_OPENVINO_NPU_KEEP_Q4_0`     | Boolean   | `0`        | NPU-only experimental toggle. Keep native `Q4_0` weights instead of regrouping them through the NPU requant policy. Can avoid requantization work and extra scale storage changes, but the resulting group-32 graph may be unstable on current NPU drivers. |
 | `GGML_OPENVINO_MANUAL_GQA_ATTN`   | Boolean   | device-based | Tri-state. When **unset**, manual GQA attention is enabled by default on `GPU` and disabled on other devices. Set to a positive integer to force-enable, or `0` to force-disable. |
 | `GGML_OPENVINO_MEMORY_OPTIMIZE`   | Boolean   | `0`        | Umbrella switch for compile-time memory reductions. Enables `GGML_OPENVINO_REDUCE_COMPILE_MEM` and, on GPU, `GGML_OPENVINO_RELEASE_WEIGHTS` unless those fine-grained variables are explicitly set. |
 | `GGML_OPENVINO_REDUCE_COMPILE_MEM`| Boolean   | inherits from `GGML_OPENVINO_MEMORY_OPTIMIZE` | Reduce compile-time host memory use by streaming weight requantization and avoiding extra weight-node materialization where possible. Set explicitly to override the umbrella switch. |
 | `GGML_OPENVINO_RELEASE_WEIGHTS`   | Boolean   | inherits from `GGML_OPENVINO_MEMORY_OPTIMIZE` on GPU | GPU-only. Release host weight buffers after the compiled model cache can reuse the device/plugin copy. Requires stable graph shapes; dynamic workloads that need recompilation should leave this disabled. |
+| `GGML_OPENVINO_SPILL_DIR`         | String    | `not set`  | Directory for a disk-backed weight buffer. When set, the repacked weight buffer is mapped from an unlinked file on this path instead of anonymous memory, so its pages are reclaimable under memory pressure instead of staying pinned, cutting the load-time host memory peak. Must point at real storage; a tmpfs mount (e.g. `/tmp` on many systems) backs it with RAM and makes the peak worse. |
+| `GGML_OPENVINO_REQUANT_KQUANT`    | String    | `not set`  | Requantize Q6_K/Q5_K weights (and matching MoE expert weights) to a 4-bit target instead of the default Q8_0_C, trading accuracy for less memory traffic. One of `q4_sym128` (Q6_K/Q5_K only), `q4_sym128_all` (Q4_K too, drops its per-group zero point), `q4_asym64_all` (Q6_K/Q5_K/Q4_K, keeps a real zero point at group 64), or `native` (no requantization). |
 | `GGML_OPENVINO_PROFILING`         | Boolean   | `0`        | Enable execution-time profiling.                                                                            |
 | `GGML_OPENVINO_DUMP_CGRAPH`       | Boolean   | `0`        | Dump the GGML compute graph to `cgraph_ov.txt`.                                                             |
 | `GGML_OPENVINO_DUMP_IR`           | Boolean   | `0`        | Serialize OpenVINO IR files with timestamps.                                                                |