Repository navigation

Fix llama-server and llama-bench - #11

Closed

wine99 wants to merge 168 commits into

dev_backend_openvinofrom

reset_variable_state

Closed

Fix llama-server and llama-bench#11
wine99 wants to merge 168 commits into
dev_backend_openvinofrom
reset_variable_state

Commits on Oct 15, 2025
- Add ggml-openvino base files
  
  YangleiZouIntel
  authored and
  wine99
  committed
  675085b
- add openvino as optional backend for Llama.cpp ggml
  
  zhanmyz
  authored and
  wine99
  committed
  5b28b4f
- * Configure the device(default CPU) that uses OpenVINO to compile the model
  
  zhanmyz
  authored and
  wine99
  committed
  8a54dfd
- Solve the issue of abnormal model output caused by using OpenVINO ADD operator
  
  zhanmyz
  authored and
  wine99
  committed
  0fc7124
- Add OpenVINO MUL operator to GGML of Llama.cpp.
  
  zhanmyz
  authored and
  wine99
  committed
  15b6a4f
- Add compile options
  
  zhanmyz
  authored and
  wine99
  committed
  f246d24
- add OpenVINO frontend convert process steps
  
  zhanmyz
  authored and
  wine99
  committed
  f7189b5
- add get openvino available ops function
  
  zhanmyz
  authored and
  wine99
  committed
  d2a306f
- Add PoC of integration of openvino frontend. Main changes: ggml-ov-frontend-utils, GraphIterator, Decoder
  
  yumengbo
  authored and
  wine99
  committed
  746bd53
- Implement GgmlOvDecoder. Add dump functions.
  
  yumengbo
  authored and
  wine99
  committed
  047a771
- Convert subgraph with add, sub, mul, div op to ov model and do infer on openvino device
  
  yumengbo
  authored and
  wine99
  committed
  21aaa5f
- Add GGML_OV_FRONTEND option. Add readme.
  
  yumengbo
  authored and
  wine99
  committed
  d3f5b62
- Change output for infer request to set output tensor. Support scale, view op.
  
  yumengbo
  authored and
  wine99
  committed
  eedde64
- add GET_ROWS operator of OpenVINO to GGML of llama.cpp
  
  zhanmyz
  authored and
  wine99
  committed
  6ee5ee4
- Update build.md and add operation mapping(GGML to OpenVINO)
  
  zhanmyz
  authored and
  wine99
  committed
  0515ab8
- add the rms_norm operator implemented using OpenVINO to the GGML backend of llama.cpp
  
  zhanmyz
  authored and
  wine99
  committed
  63327ea
- Fix issue for output memory copy of infer request
  
  yumengbo
  authored and
  wine99
  committed
  8037eb3
- Change to implementation following pytorch frontend
  
  yumengbo
  authored and
  wine99
  committed
  b111e6a
- Add support for UNARY SILU op . Fix pytorch impl bugs.
  
  yumengbo
  authored and
  wine99
  committed
  4b91d1f
- Support Softmax op
  
  yumengbo
  authored and
  wine99
  committed
  923166f
- Support Softmax op
  
  yumengbo
  authored and
  wine99
  committed
  f783c2a
- Support ROPE op.
  
  yumengbo
  authored and
  wine99
  committed
  aedb88a
- Add support for RMS_NORM OP
  
  zhanmyz
  authored and
  wine99
  committed
  67d51bd
- Add MUL_MAT,CPY,CONT as operators implemented in OpenVINO for GGML backend
  
  zhanmyz
  authored and
  wine99
  committed
  33cf85f
- Move CPY from GGML OV Backend to OV Frontend
  
  zhanmyz
  authored and
  wine99
  committed
  cba097b
- add implementation of MUL_MAT, CPY, CONT of GGML ops using OV ops
  
  zhanmyz
  authored and
  wine99
  committed
  9b74f0f
- add implementation of CPY when the output tensor is non-contiguous
  
  zhanmyz
  authored and
  wine99
  committed
  68d53bb
- add tmp source code files
  
  zhanmyz
  authored and
  wine99
  committed
  1dc4ec6
- Execute singel CONT operator is OK
  
  zhanmyz
  authored and
  wine99
  committed
  d941a56
- Execute CONT & VIEW operators in OV Frontend is OK
  
  zhanmyz
  authored and
  wine99
  committed
  f670c64
- OV Frontend supports GET_ROWS/RMS_NORM/MUL/MUL_MAT graph conversion of consecutive OPs
  
  zhanmyz
  authored and
  wine99
  committed
  1fa50c9
- OV Frontend supports GET_ROWS/RMS_NORM/MUL/MUL_MAT/ROPE/SCALE/SOFTMAX/ADD adjacent op graph conversion
  
  zhanmyz
  authored and
  wine99
  committed
  5c3face
- Change the input parameter shape of CONT operator
  
  zhanmyz
  authored and
  wine99
  committed
  e986929
- Change the input and ouput node shape of MUL_MAT operator
  
  zhanmyz
  authored and
  wine99
  committed
  b4aad38
- Change the input and ouput node shape of MUL_MAT operator
  
  zhanmyz
  authored and
  wine99
  committed
  0b716bb
- change CONT and MULMAT input node shape
  
  zhanmyz
  authored and
  wine99
  committed
  ba490a6
- All adjacent ops can conversion but calculation result is wrong and need debugging
  
  zhanmyz
  authored and
  wine99
  committed
  55ddb14
- 1. All operators implemented using OpenVINO can be successfully executed individually.
  
  zhanmyz
  authored and
  wine99
  committed
  7c49c79
- 1. Update the implementation of CPY node when it's non-contiguous
  
  zhanmyz
  authored and
  wine99
  committed
  7eb288b
- Minor Update
  
  zhanmyz
  authored and
  wine99
  committed
  137096b
- Try to add VIEW node to OV Frontend and have some issues that need to be dealt with
  
  zhanmyz
  authored and
  wine99
  committed
  d42f931
- 1. In the Prompt process and predict first token stage, the PERMUTE node needs to be integrated into the OV Frontend
  
  zhanmyz
  authored and
  wine99
  committed
  f50d073
- add debug info
  
  zhanmyz
  authored and
  wine99
  committed
  53dd566
- Process Prompt and predict first token is OK
  
  zhanmyz
  authored and
  wine99
  committed
  f951dae
- 1. Solve the AC issue of Permute+VIEW and MULMAL issue in the phase of “1. Process Prompt and predict the first token”.
  
  zhanmyz
  authored and
  wine99
  committed
  a2f9006
- 1. Delete some comments
  
  zhanmyz
  authored and
  wine99
  committed
  1d61ab0
- * Use find_package in CMake to configure OpenVINO
  wine99
  committed
  987d9ee
- change op mappings to list in openvino_supports_op
  wine99
  committed
  c2768e2
- 2nd+ token correct by fix CPY in OV, remove single op backend compute code
  wine99
  committed
  0e9e6f0
- Arbitrary token len (>32) work; Fix bug in mulmat
  wine99
  committed
  2486762
- FEAT: do PERMUTE eagerly
  wine99
  committed
  089e253
- FEAT: Add interleaved mode for ROPE
  wine99
  committed
  2fd34a6
- REFACTOR: support weigts as constant
  wine99
  committed
  debb3dc
- STYLE: minor refactor
  wine99
  committed
  3fcb8e2
- PERF: share const nodes for weights for diff infer
  wine99
  committed
  33a2c08
- BUILD: update build doc, add cmake preset, add CACHE_DIR env var
  wine99
  committed
  d63506e
- FEAT: improve debug capability
  wine99
  committed
  78f63af
- PERF: compile once (dynamic graph + cache)
  wine99
  committed
  12b5c86
- Rebase - Bring up to date and fix build process
  
  virajwad
  authored and
  wine99
  committed
  d38e14a
- fix build error
  wine99
  committed
  3c07b5a
- FIX: backend buffer type issue
  wine99
  committed
  3efe972
- STYLE: clang-format
  wine99
  committed
  985510a
- FEAT: Add all conversion code from ov side
  wine99
  committed
  c18d1d3
- PERF: favor low precision matmul
  wine99
  committed
  bbe9024
- STYLE and minor REFACTOR
  wine99
  committed
  3a2ce74
- FIX: Re-add tensor names in cgraph, Add another case for RESHAPE
  wine99
  committed
  7335069
- FIX: input shape of KQ_mask
  wine99
  committed
  94b25eb
- PERF: add weight constant in parallel
  wine99
  committed
  17fdbee
- FIX: set_max_token_len
  wine99
  committed
  9219181
- PERF: use Slice+Concat in writing cache_v
  wine99
  committed
  fa4e63a
- Update build doc
  wine99
  committed
  9bd970d
- Add cgraph tensor output name to OV op name
  wine99
  committed
  0f5cca4
- Update openvino build instructions
  
  ravi9
  authored and
  wine99
  committed
  59ca4b5
- Add initial NPU support
  wine99
  committed
  5abccbe
- draft NPU support version 2: prefill + kvcache
  wine99
  committed
  f7f13b6
- NPU support version 2: prefill + kvcache
  wine99
  committed
  9256f60
- Change due to ggml cgraph changes, not correct yet
  wine99
  committed
  4968ddb
- Change due to ggml cgraph changes, llama-3.2 CPU work
  wine99
  committed
  f4ead6d
- Add AMD64 to CMakeLists
  wine99
  committed
  b498444
- Change due to ggml cgraph changes, all device work
  wine99
  committed
  477c289
- Refactor: clean, fix warning
  wine99
  committed
  36acacd
- Update clang-format
  wine99
  committed
  f0e732b
- Statful transformation for CPU GPU
  wine99
  committed
  85a10a1
- Add SwiGLU
  wine99
  committed
  44e9dcc
- Fuse to SDPA
  wine99
  committed
  bc12dc1
- Replace Concat with Broadcast in MulMat for GQA
  wine99
  committed
  35249d0
- Pull out indices creation for kv cache update
  wine99
  committed
  5a78d7f
- Refactor: remove past_token_len from extra_inputs
  wine99
  committed
  0c6bae4
- Fix Phi3 SwiGLU and SoftMax
  wine99
  committed
  91b3c15
- Pull out sin cos from rope
  wine99
  committed
  94ca8fd
- Reduce memory: free ov weights node after graph conversion
  wine99
  committed
  5a95352
- Fix CPY due to cgraph change
  wine99
  committed
  a15fe5f
- Added OpenVINO CI/CD. Updated docs
  
  ravi9
  authored and
  wine99
  committed
  b1ccdd9
- Fix llama-cli
  wine99
  committed
  62695f0
- Fix Phi3 ROPE; Add test-backend-ops
  wine99
  committed
  7c309ae
- Fix NPU
  wine99
  committed
  5b37307
- Fix llama-bench; Clang-format
  wine99
  committed
  9536b84
- Fix llama-perplexity
  wine99
  committed
  c3c2960
- temp. changes for mark decomp
  
  cavusmustafa
  authored and
  wine99
  committed
  0a207ec
- matmul in fp32
  wine99
  committed
  8b1bf53
- mulmat input conversion fix
  
  cavusmustafa
  authored and
  wine99
  committed
  8ed93e9
- mulmat type conversion update
  
  cavusmustafa
  authored and
  wine99
  committed
  06213b1
- add mark decomp pass
  
  cavusmustafa
  authored and
  wine99
  committed
  a7bc9dd
- Revert changes in fuse_to_sdpa
  wine99
  committed
  3b1639f
- Update build.md
  
  ravi9
  authored and
  wine99
  committed
  338fd25
- Fix test-backend-ops
  wine99
  committed
  6a1b424
- Skip test-thread-safety; Run ctest only in ci/run.sh
  wine99
  committed
  9351310
- Use CiD for NPU
  wine99
  committed
  57e92d2
- Optimize tensor conversion, improve TTFT
  wine99
  committed
  ec8571e
- Support op SET_ROWS
  wine99
  committed
  01f0a3e
- Fix NPU
  wine99
  committed
  bcb173c
- Remove CPY
  wine99
  committed
  179d68f
- Fix test-backend-ops
  wine99
  committed
  ebc5c33
- Minor updates for raising PR
  wine99
  committed
  b26f6bb
- Perf: RMS fused to OV internal RMS op
  wine99
  committed
  9d6526f
- Fix after rebasing
  wine99
  committed
  0136338
- Change openvino device_type to GPU; Enable flash_attn
  wine99
  committed
  48c5cc5
- Update supports_buft and supports_op for quantized models
  wine99
  committed
  43c852a
- Add quant weight conversion functions from genai gguf reader
  wine99
  committed
  db049d0
- Quant models run with accuracy issue
  wine99
  committed
  0caad15
- Fix accuracy: disable cpu_repack
  wine99
  committed
  970eca1
- Fix CI; Disable test-backend-ops
  wine99
  committed
  7186a8b
- Fix Q4_1
  wine99
  committed
  5ace942
- Fix test-thread-safety
  wine99
  committed
  5029367
- Fix test-backend-ops: Treat quantized tensors as weights
  wine99
  committed
  a262e21
- Add NPU Q4_0 support
  wine99
  committed
  dea1d2a
- NPU perf: eliminate zp
  wine99
  committed
  11e289c
- Dequantize q4_1 q4_k q6_k for NPU
  wine99
  committed
  7b64a16
- Add custom quant type: q8_1_c, q4_0_128
  wine99
  committed
  ffca963
- Set m_is_static=false as default in decoder
  wine99
  committed
  9459bc5
- Simpilfy translation of get_rows
  wine99
  committed
  ad296b7
- Fix after rebasing
  wine99
  committed
  d9333b1
- Improve debug util; Eliminate nop ReshapeReshape
  wine99
  committed
  32206c4
- STYLE: make get_types_to_requant a function
  wine99
  committed
  9db6bb4
- Support BF16 model
  wine99
  committed
  f8e9be7
- Fix NPU compile
  wine99
  committed
  fe383f3
- WA for npu 1st token acc issue
  wine99
  committed
  5732262
- Apply EliminateZP only for npu
  wine99
  committed
  077f9fc
- Add GeGLU
  wine99
  committed
  60ec76b
- Fix Hunyuan
  wine99
  committed
  d942ac2
- Support iSWA
  wine99
  committed
  228a742
- Fix NPU accuracy
  wine99
  committed
  77da485
- Fix ROPE accuracy when freq_scale != 1
  wine99
  committed
  5938048
- Minor: not add attention_size_swa for non-swa model
  wine99
  committed
  a655aad
- Minor refactor
  wine99
  committed
  21614a1
- Add Q5_K to support phi-3-q4_k_m
  wine99
  committed
  96254f9
- Requantize Q6_K (gs16) to gs32 on GPU
  wine99
  committed
  f785c3d
- Fix after rebasing
  wine99
  committed
  e6dca1b
- Always apply Eliminate_ZP to fix GPU compile issue on some platforms
  wine99
  committed
  2413170
- kvcachefusion support
  
  cavusmustafa
  authored and
  wine99
  committed
  e444b88
- env variable GGML_OPENVINO_DISABLE_SDPA_OPTIMIZATION added
  
  cavusmustafa
  authored and
  wine99
  committed
  c40b9fe
- Fix for Phi3
  
  cavusmustafa
  authored and
  wine99
  committed
  c21d664
- Fix llama-cli (need to run with --no-warmup)
  wine99
  committed
  3370915
- Fix add_sliced_mask; Revert mulmat, softmax; Remove input attention_size, iSWA model not working
  wine99
  committed
  47843b6
- fix after rebasing
  wine99
  committed
  63f6bba
- Fix llama-3-8b and phi3-mini q4_0 NPU
  wine99
  committed
  b41f494
- Update to OV-2025.3 and CMakeLists.txt
  
  ravi9
  authored and
  wine99
  committed
  d4eb8ec
- Add OV CI cache
  wine99
  committed
  f89292d
- Apply CISC review and update CI to OV2025.3
  ravi9
  authored
  89b8212
- Update CI to run OV dep install before build
  ravi9
  authored
  09ead55
- Update OV dockerfile to use OV2025.3 and update build docs
  ravi9
  authored
  7d8ea73
Commits on Oct 21, 2025
- Style: use switch in supports_ops
  wine99
  committed
  6cac650
- Style: middle ptr and ref align, omit optional struct keyword
  wine99
  committed
  9e02b3f
- Fix llama-cli; WIP llama-server
  wine99
  committed
  ac57c17
- Fix: llama-server works with `--cache-ram 0`; llama-bench works with `-fa 1`
  wine99
  committed
  affee8d
- Minor udpates
  wine99
  committed
  886d418
- Fix llama-server
  wine99
  committed
  5f25e52
- Style: clang-format
  wine99
  committed
  1b55304

Commits

Commits on Oct 15, 2025