You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
quantized version of the model fo CPU inference #78
I wanted to run ColPali on CPU, and the GGUF model is a good choice for this. But as of now, there is no support for quantizing Paligemma models to GGUF. Would you happen to know if there is a better way to run these models on CPU?
It might not be as efficient as GGUF, but with OpenVino Executor and 8-bit quantization, it works decently on the CPU. I get about 5-8 seconds per page embedding speed on an old AMD Ryzen 7 5800x processor and about 5 seconds per 20 pages on NVIDIA 3070Ti.
Activity
ManuelFay commentedon Oct 9, 2024
I would also be super interested in looking into that but haven't yet ! Lemme know if you have interesting findings !
akshayballal95 commentedon Oct 20, 2024
With some tweaking around in the HuggingFace Optimum exporter, I was able to get an ONNX model.
https://huggingface.co/akshayballal/colpali-v1.2-merged-onnx
It might not be as efficient as GGUF, but with OpenVino Executor and 8-bit quantization, it works decently on the CPU. I get about 5-8 seconds per page embedding speed on an old AMD Ryzen 7 5800x processor and about 5 seconds per 20 pages on NVIDIA 3070Ti.
And for GGUF, I think we need to wait for ggml-org/llama.cpp#7553 to merge.