[martindevans]

Previous update (#460) was cancelled, due to some other work which changed things (introducing CLBLAST binaries, renaming windows binaries).

• Build job: https://github.com/SciSharp/LLamaSharp/actions/runs/7746303349
• llama.cpp version: https://github.com/ggerganov/llama.cpp/tree/d71ac90985854b0905e1abba778e407e17f9f887

[martindevans]

Testing:

• Windows (CPU)
• Windows (Nvidia GPU)
• Windows (AMDGPU)
• Linux (CPU)
• Linux (GPU)
• MacOS (ARM)
• MacOS (x86)

[m0nsky]

My application seems to run fine in both avx2 and cu12.1.0, however, upping the batch size from 4096 to 8192 still throws:

System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.

when using or cu12.1.0 (while avx2 has no issues with 8192 batch size). I don't think I'll need a 8192 batch size and I'm not sure if this is expected, but just wanted to let you know.

Also, clblast throws a LLama.Exceptions.RuntimeError: The native library cannot be correctly loaded. for me. Does it require anything else to be installed?

Edit
I'm on Windows 10, RTX 3080

[martindevans]

@jasoncouture any ideas about the CLBLAST loading issue?

[zsogitbe]

I have tested the new code with several models on Windows 10 & GPU and it works fine. Thank you Martin!
The above errors are most likely errors in llame.cpp. Try another version. I remember having issues like this with some versions of the C++ code. Also, it seems to me that they have corrected a bug in the batch processing code just yesterday.

[m0nsky]

@martindevans Got clblast to work. I needed to copy clblast.dll to my executable dir (alongside the existing llama.dll), I just grabbed it from the latest llama.cpp release for now.

[martindevans]

Thanks for testing that @m0nsky, I've added a note to #464 (which is the tracking issue for OpenCL support) about distributing clblast.dll in the nuget packages.

[ladeak]

Is there a preview nuget package to be tested?

[martindevans]

@ladeak there's no preview package, if you want to test it your best option is to clone this branch

[jasoncouture]

@martindevans should we use something like nbgv and publish preview builds?

[ladeak]

@martindevans I was able to test Phi-2 on Windows 11 x64. However, I notice there is a breaking change in the sense if I want to load a model for the LLamaEmbedder and for the LLamaSharpChatCompletion I would need to load it with EmbeddingMode set to true. Is that right?

EDIT: I tested through the semantic-kernel integration.

[martindevans]

I notice there is a breaking change in the sense if I want to load a model for the LLamaEmbedder and for the LLamaSharpChatCompletion I would need to load it with EmbeddingMode set to true. Is that right?

Yep that's a deliberate change 👍

[jasoncouture]

@martindevans I posted the PRs for full CLBLAST including the nuspec, see #489 , solves the issues of the missing clblast.dll, and adds linux AMDGPU support as well.

[martindevans]

New run started here, using the same commit version.

[jasoncouture]

@martindevans looks like it failed due to that sha256 check. I'll put up a PR to pull it for now, and get it figured out.

[martindevans]

Started another run here

[ladeak]

When do you plan to release the next version?

[martindevans]

@jasoncouture is doing some tweaks to CLBLAST support, I think if that's done soon we'll probably release then.

[liqngliz]

Any update when this release will be out?

[martindevans]

When it's ready! Hopefully this week.

I think if it's not done soon we'll push CLBLAST to the next release and get everything else out the door :)