Repository navigation
Only Distribute CUDA Binaries? #298
Description
As of this PR to llama.cpp the CUDA binaries are capable of running with CPU only, as long as n_gpu_layers = 0.
This might mean that we can significantly simplify our distribution of binaries by removing the CPU only variants and only shiping CUDA ones.
Activity
martindevans commentedon Nov 15, 2023
There's some further discussion over in the linked PR with some potential issues that may make this less attractive:
SanftMonster commentedon Nov 17, 2023
It's a good news to hear this feature supported by llama.cpp. However, I have some disagreements.
If the user does not have the cuda installed, using cuda backend makes little sense. Besides, the cpu backend is much smaller than the cuda backend, which may be desired in some conditions. I prefer to keep separating these two backends. :)