Skip to content
This repository was archived by the owner on Jun 5, 2025. It is now read-only.
This repository was archived by the owner on Jun 5, 2025. It is now read-only.

Ollama support #70

Closed
1 of 1 issue completed
Closed

Description

@lukehinds

Based on the findings of @ptelang , docker is not able tap Apple M1/2/3 based GPUs. However with llama.cpp now implementing a very-fast CPU-accelerated Q4_0_4_4 quantized inference inference of mode sub 5 billion parameters models seems quite viable.

However its quite possible some will still want to use OLLama for running very large models, or simply because it is there prefered way of managing multiple models. We should therefore extend inference support towards Ollama.

Activity

  1. added this to the 0.1.0 (flagship) milestone on Nov 22, 2024
  2. self-assigned this
    on Nov 22, 2024
  3. added a commit that references this issue on Dec 1, 2024
    34ca65c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions