Skip to content
MaxusAI
ollama
Public
forked from ollama/ollama

About

A fork of ollama/ollama adding vision-model correctness fixes — image token budgets, dynamic resolution, and an fp16-overflow gate for qwen2.5-vl — plus a vision regression suite. Tracks upstream releases; fixes are offered upstream and dropped here once they land.

ollama.com

Resources

Readme
License

Contributing

Contributing

Security policy

Security policy
Activity
Custom properties

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository
BranchesTags
 
 

Latest commit

 

History

7,094 Commits

Folders and files

NameName
Last commit message
Last commit date
.claude/skills
.claude/skills
 
 
.github
.github
 
 
anthropic
anthropic
 
 
api
api
 
 
app
app
 
 
auth
auth
 
 
cmake
cmake
 
 
cmd
cmd
 
 
create
create
 
 
decision
decision
 
 
discover
discover
 
 
docs
docs
 
 
envconfig
envconfig
 
 
format
format
 
 
fs
fs
 
 
harmony
harmony
 
 
integration
integration
 
 
internal
internal
 
 
llama
llama
 
 
llm
llm
 
 
logutil
logutil
 
 
manifest
manifest
 
 
middleware
middleware
 
 
ml
ml
 
 
mlx
mlx
 
 
mlxrunner
mlxrunner
 
 
model
model
 
 
openai
openai
 
 
parser
parser
 
 
progress
progress
 
 
readline
readline
 
 
scripts
scripts
 
 
server
server
 
 
template
template
 
 
thinking
thinking
 
 
tools
tools
 
 
transfer
transfer
 
 
types
types
 
 
version
version
 
 
.dockerignore
.dockerignore
 
 
.gitattributes
.gitattributes
 
 
.gitignore
.gitignore
 
 
.golangci.yaml
.golangci.yaml
 
 
AGENTS.md
AGENTS.md
 
 
CLAUDE.md
CLAUDE.md
 
 
CMakeLists.txt
CMakeLists.txt
 
 
CMakePresets.json
CMakePresets.json
 
 
CONTRIBUTING.md
CONTRIBUTING.md
 
 
Dockerfile
Dockerfile
 
 
Dockerfile.applearm
Dockerfile.applearm
 
 
Dockerfile.gemma4budget
Dockerfile.gemma4budget
 
 
Dockerfile.rocm
Dockerfile.rocm
 
 
LICENSE
LICENSE
 
 
LLAMA_CPP_VERSION
LLAMA_CPP_VERSION
 
 
MLX_C_VERSION
MLX_C_VERSION
 
 
MLX_VERSION
MLX_VERSION
 
 
README.md
README.md
 
 
SECURITY.md
SECURITY.md
 
 
go.mod
go.mod
 
 
go.sum
go.sum
 
 
main.go
main.go
 
 

Repository files navigation

This is a fork of ollama/ollama

Everything below this banner is upstream's README, describing upstream's work. Ollama is theirs; this fork only adds to it, and tries to add as little as possible.

It tracks upstream releases and carries a narrow set of changes:

  • Vision-model correctness. Image token budgets and native-aspect dynamic resolution for gemma4 and nemotron, and an fp32-accumulation gate for qwen2.5-vl that closes an fp16 overflow in the vision tower — a few elements in millions reach inf on CUDA and the caption collapses into a repeated glyph. Offered upstream as ollama#18070.
  • A vision regression suite. A preflight harness with recorded per-model expectations, generated (public, no-download) trigger images, and env-gated node-level instrumentation, run against every build before it deploys.
  • Fixes carried until upstream takes them, each tracked against an upstream issue or PR, and deleted from here when it lands there — the retirement register lists every carried item, what retires it, and the test that gates its deletion.
  • An experimental MLX runtime for Apple Silicon and CUDA — see the caveats below before using it for anything that matters.

Current fold: v0.34.4-dynres — upstream v0.34.4, llama.cpp b11081 with compat patches 001 002 004 005 801 802 903 and 908 (908 is new in this fold), MLX 59d600b5, XGrammar 0.2.7. main moves ahead of this between folds; the tag is the fixed point to build and roll back to. Deployed: the tag, stamped 0.34.4-dynres-0-gb43ee8e. It went onto the CUDA host at 07:37 on 2026-09-28, onto the AMD/gfx1151 host at 07:39 (#391), and onto the Apple Silicon host's mlx-metal surface (:11435) at 14:07.

  • Build: on the CUDA and gfx1151 hosts, the natively gated payload with the tag's Go binary. On the Apple Silicon host, both halves are built at the tag.
  • think+format: all three hosts run the two-pass flow (OLLAMA_FORMAT_TWO_PASS=1, ADR 0045; open item 7 of the fold record).
  • KV cache: all three set OLLAMA_KV_CACHE_TYPE=f16 explicitly (#387).
  • Drafting: the CUDA container and the Apple Silicon launchd agent also keep OLLAMA_MLX_DRAFT_UNDER_GRAMMAR=0 (ADR 0033).

The matrix below is each host's preflight on its production server. On the Apple Silicon host, the tag replaced 0.34.0-maxusai-8a7ba949, which had served the mlx-metal surface on :11435 since 2026-09-18. Before the swap, the tag's build passed preflight on a :11437 stage, the mlxrunner tests with vision golden parity, and a two-pass smoke. Both builds' binary and payload are archived for rollback (BINARIES.md).

Fork builds are stamped <upstream-version>-dynres-<n>-g<sha>; dynres names the change that started the fork, not the company that runs it. Fork-specific documentation, ADRs and measurements live in docs/maxusai/.

What tested green for this fold

surface Build identity M5 tensor path Image size ladder Pinned image budget thinking on/off Output quality fp16 overflow canary Runner isolation measured on
cuda green n/a green green green not run green green 0.34.4-dynres-0-gb43ee8e
mlx-cuda not run not run not run not run not run not run not run not run —
mlx-metal skipped green green skipped green not run skipped green 0.34.4-dynres-0-gb43ee8e
apple-silicon-mlx not run not run not run not run not run not run not run not run —
rocm7 green n/a green green green green green green 0.34.4-dynres-0-gb43ee8e
rocm10 not run not run not run not run not run not run not run not run —
cpu not run not run not run not run not run not run not run not run —

Generated by release_matrix.py from recorded preflight runs. A surface with no run for this release reads not run — absence is shown, never assumed green. A group is reported at its weakest check, so one skipped probe does not read as a pass.

cuda, mlx-metal and rocm7 here are the deployed serving surfaces. apple-silicon-mlx is the deprecated alias for mlx-metal and will disappear from the generator with it.

What differs from upstream, concretely

Measured against upstream ollama v0.34.4 at llama.cpp b11081. Every row is a capability the fork has and upstream does not; the record column is where the decision and its measurements live (docs/maxusai/). The retirement register tracks when each one can go.

Vision correctness on the llama.cpp path — the deployed engine

upstream ollama this fork record
nemotron-3 vision fixed 512×512 canvas — 256 tokens per image, whatever the aspect ratio. Upstream's MLX nemotron_h has been native-aspect since v0.34.3 native-aspect dynamic resolution, 256–3,328 tokens, position embeddings interpolated to the patch grid in-graph patch 002, ADR 0001
gemma4 image budget default limits 70–1,120 tokens (40–280 before b10864); an under-budget image keeps its natural rounded grid and is letterbox-padded every image scaled to fill the requested budget and snapped to gemma4's supported ladder (70/140/280/560/1120), never padded — off-ladder grids measurably break box_2d vertical grounding. The budget is a per-request option (image_min_tokens/image_max_tokens, defaults 70/1120) and the scheduler reloads when the resolved flags change. Upstream has since adopted the same default limits; the fill is still fork-only patch 004, ADR 0003/0008/0016
qwen2.5-vl on CUDA f16 vision matmuls accumulate in fp16; on some ordinary images a few elements of millions reach inf at v.blk.31.ffn_down and the caption collapses into one repeated glyph fp32 accumulation forced for every Qwen-VL runner (qwen2vl and qwen25vl, one family under two converter spellings), keyed on the GGUF architecture. Offered upstream as ollama#18070, still open llm/llama_server.go
MoE + MMQ on CUDA MMQ reads its buffers in whole tiles, but pads them from the batch (get_J_max(ne12) since #29941, get_J_max(ne11) before), which rounds down while the launched tile rounds up. Before #29941, a broadcast ne11 == 1 gave no padding at all: an illegal memory access. Since, batches under 128 tokens can still read past the buffer every tile-read buffer (src1, ids_dst, NVFP4 scales) padded for the widest tile that has a config, as before llama.cpp #24127. Offered as llama.cpp#27044, which upstream closed for #29941 patch 903
gemma4 flash attention on CUDA b11081 retunes the MMA configs and tile sizes for head dims 256/512 (ce8caa6e6). Its accuracy is unchanged in llama.cpp's test-backend-ops, but the rounding moves. With ollama's Q4_K_M, gemma4:26b leaves 6 of 27 think-on cases in loops that never end, against 1 without it; with ggml-org's Q4_0, a case loops the other way the device half of ce8caa6e6 is reverted to b10969's tiling, which keeps the numerics production was gated on; the host half, the decode selection, stays upstream's. The revert shows no speed cost: in the paired runs on CUDA, gemma4:31b decoded 57 tok/s with either tiling, and gemma4:26b 176 tok/s against the new tiling's 137. On gfx1151 the patch changes no kernel patch 908, the v0.34.4 fold record

Structured output and generation control

upstream ollama this fork record
think + format in one request since v0.34.4, one pass: the grammar applies from the first token, after free thinking that ends at the parser's closing strings the same by default. OLLAMA_FORMAT_TWO_PASS=1 keeps the fork's two-pass flow: pass one thinks without the grammar and stops at the think-close marker, and pass two answers under it. Production runs it, because on MLX it lets the thinking draft ADR 0045 (0002/0004/0010)
whitespace in structured output (MLX) the JSON grammar allows an unlimited whitespace run between tokens, so a stalled decode can fill num_predict with indentation and never close the answer runs bounded at 32 characters per separator (max_whitespace_cnt on the json_schema element) ADR 0035 (proposed)
drafting under a grammar (MLX) always on on by default to match upstream; OLLAMA_MLX_DRAFT_UNDER_GRAMMAR=0 restores the gate, and production sets it ADR 0033
stop sequences (MLX) not honoured by the MLX runner honoured, with a possible stop prefix held back until it matches or the stream ends mlxrunner/stopper.go
KV cache type one global OLLAMA_KV_CACHE_TYPE per model, with K/V pair syntax; production runs f16 on every platform, and a quantized cache is per model or per request ADR 0005, ADR 0043
llama-server's prompt cache no --cache-ram, so llama.cpp's 8192 MiB host-RAM cache is on for every GGUF model the same by default. prompt_cache_ram, per model or per request, sets --cache-ram, and 0 turns it off: up to a fifth less time per image request on gemma4:31b, with the same answers. The vision harness sends 0 ADR 0047

What production's choices cost, against stock v0.34.4

  • Two-pass think+format (OLLAMA_FORMAT_TWO_PASS=1, which production sets):
    • A request prefills twice: pass two re-submits pass one's tokens.
    • num_predict bounds each pass, so the total can exceed it (8,290 against 8,192, measured).
    • It also handles a tool call after the thinking, and an end of turn inside it, differently. ADR 0045 tabulates both flows.
    • On MLX the thinking runs as fast as a single pass that drafts under the grammar, as stock's does (level within the spread, measured on the fork's build with drafting on). Production turns drafting under a grammar off (OLLAMA_MLX_DRAFT_UNDER_GRAMMAR=0), because it retains memory on the qwen3.5 family, and drafts in pass one, which has no grammar.
    • On GGUF neither flow drafts. The two flows leave the same cases unfinished, on all five models measured on gfx1151.
  • The MLX runtime is slower than the llama.cpp engine on CUDA; see below.
  • Patch 908 costs no speed (the table above), and an f16 KV cache is stock's default too.

Serving and scheduling

upstream ollama this fork record
MLX admission weights against free device memory (and, since v0.34.1, a system-memory bound on integrated GPUs) weights + KV priced at the requested num_ctx + a per-architecture headroom; an explicit rung that does not fit is refused, an automatic one is clamped ADR 0034
MLX memory ceiling none OLLAMA_MLX_MEMORY_LIMIT and a cache limit, set per runner from the admitted budget runner knobs
nvfp4 global scales stored in MLX's m × 2688 form and divided back out wherever a wrapper applies the scale itself (globalScaleFactor, ollama#18550). That is not the identity in float32: 17 of gemma4:31b's 191 vision scales move one ulp stored as the checkpoint's own multiplier m, with upstream's helpers defined in those terms, so every call site is exact ADR 0039
model identity in a record a tag the manifest digest: the library re-published gemma4:*-nvfp4 with bf16 vision towers under unchanged tags and config blobs ADR 0038 (proposed)
gemma4 image chunk vs. generation batch (GGUF) the batch follows num_ctx (1024 above 4096), so a top-rung gemma4 image (up to 1120 tokens) is decoded in two pieces, bidirectional only within each. The fix that fits an image chunk to one ubatch, llama.cpp#28954, is still open a gemma4 vision runner starts from the batch rung that holds its image ceiling (2048 at 1120) and steps down only when it does not fit ADR 0036
gemma4 on MLX upstream's own vision and audio tower. Since v0.34.4 it picks each image's budget from the 70/140/280/560/1120 ladder, closest to the input resolution, with no API parameter (ollama#18603) vision through the fork's pipeline on upstream's MediaModel, with the GGUF path's per-request budget (image_min_tokens/image_max_tokens, defaults 70/1120), filled and snapped to the ladder; audio not shipped ADR 0021 (0003/0008)
transparent images (gemma4 on MLX) alpha dropped by RGB conversion before the resize, so the colour stored under a transparent pixel shows composited over white before the resize, as mlx-vlm's convert_to_rgb does ADR 0015
media prompts on MLX chunk boundaries extended around non-causal spans (extendChunk, since v0.34.1) prefill chunks span-aligned around image blocks, the opening chunk included, with bidirectional expansions matched all-or-nothing; a late image is refused ADR 0014
scheduler — log sites never drop fields under contention; head-of-line and evict-all-wait fixes; attached media charged against capabilities before the load; capability advertising corrected for MLX architectures server/sched.go, images.go
panic hygiene (MLX) — a cleanup that fails while a request is unwinding never replaces the panic that caused it mlxrunner/unwind.go

Measurement — nothing comparable upstream

this fork record
vision regression suite preflight with versioned per-model expectations, generated (public, no-download) trigger images, an env-gated node-level meter, and a generated release matrix — run before every deploy ADR 0011/0012, patch 801
campaigns five report templates rendered only by generators; per-request memory and drafting analysis; bbox conformance scoped to image geometry ADR 0012/0028/0030
bounding-box protocol requests pin norm-1000 and carry a self-calibrating anchor, so the model's internal resize cannot contaminate coordinates: 111 of 112 cells convert cleanly across 14 geometries × 4 models × 2 think modes. A protocol and its measurements, not a runtime change ADR 0027/0030
fork identity builds stamped <upstream>-dynres-<n>-g<sha>, a tag per fold, release notes carrying the generated matrix ADR 0032

MLX runtime — experimental, and slower on CUDA

It works. Models load, stay resident and generate correct output on both Metal and CUDA. But it is not the path to reach for by default:

  • On CUDA it runs at 34–75% of the cuda path's decode throughput, median 46%. Measured across four matched model pairs on one host, one server process — full report. "Roughly half" is a fair central estimate and a poor description of any single case: the spread is 2.2× and it is not architectural (the two dense pairs sit at 75% and 53%, the two MoE at 34% and 39%). Two of the four mlx-cuda arms had no stable throughput to quote at all. The report is dated: it measured 0.33.2-dynres-5-g2b95b4a on 2026-08-30. The MLX pin has moved since then, to 59d600b5 in v0.34.4, and the matched comparison has not been re-run.

  • Its bigger cost is variance, not speed.

    • Throughput. mlx-cuda's per-request spread is ~5× the cuda path's and reaches 46% within a single arm — same host, same prompt, back to back — while every cuda arm held inside ±1.5% first time. It is per-model: gemma4:31b-nvfp4 reproduced to 1.1% across four independent measurements.
    • Output. At temperature 0 a drafted request takes its own path every time. Even undrafted, each cold load takes its own greedy trajectory. The same gemma4:26b request finished 1 to 3 times in 5 cold loads, depending on the prompt. On the cuda path, the same request repeats byte for byte (v0.34.4 fold record: the drafting probe, and open item 8).

    For anything that sets a timeout or compares two builds, this matters more than the ratio. Compare MLX builds with repeats, never single runs.

  • Structured output with thinking depends on drafting. Such a request thinks 1.5–1.7× faster when its thinking drafts. Production runs the two-pass flow, which lets it draft (ADR 0045).

  • Two traps, both measured:

    • Pin num_ctx. Unset, the default is derived from total VRAM: 262144 on the CUDA host's 105.5 GiB. That collapses decode 25×, to 1.48 tok/s against 37.97 at 8192 (record).
    • Persist the kernel cache. Without MLX_PTX_CACHE_DIR, every fresh container recompiles MLX's JIT kernels, so its first request takes 10–15 minutes. Keep one cache directory per GPU architecture (vision-suite README).

  • On Metal it is the other way round. A matched campaign measured MLX ~2.4× faster than llama-server (gemma4 12b: 121 vs 50 tok/s decode; 2026-08-08, single samples, directional only). So the CUDA gap is CUDA-specific, not an MLX property — do not generalise either number to the other platform.

  • Engine and quantization move together in every figure above — nvfp4 on MLX against q4_K_M on GGUF, because those are the artefacts that exist. So the throughput numbers describe the two stacks as shipped, not the engine in isolation, and the same confound makes MLX-vs-GGUF quality an uncontrolled comparison: a quality difference cannot be attributed to the engine either. Nobody has separated them; until someone does, treat "which is better" as open.

  • It is converging with upstream's own MLX work, and is expected to be superseded by it.

    • The fork has retired its constrained-sampling layer in favour of upstream's engine (ADR 0033).
    • Since v0.34.4 it follows upstream's single-pass structured outputs by default (ADR 0045).
    • What remains fork-only on MLX is in the tables above.

Use GGML/llama-server for anything where throughput or comparability matters.

Every row in the tables above is a delta we would rather not have. Each is offered upstream where it is upstream's to take, and deleted from here once it lands there. The Qwen-VL accumulation gate (ollama#18070) is filed and pending. Upstream closed the MMQ padding fix (llama.cpp#27044) for its narrower #29941, which still reads past the buffer below 128 tokens. 903 now carries the widest-tile padding that a successor would offer. The gemma4 tiling revert (908) is not upstream's to take: ce8caa6e6 loses no precision in llama.cpp's test-backend-ops, so the revert is the fork's own choice to keep production's gated numerics.

ollama

Ollama

Start building with open models.

Download

macOS

curl -fsSL https://ollama.com/install.sh | sh

or download manually

Windows

irm https://ollama.com/install.ps1 | iex

or download manually

Linux

curl -fsSL https://ollama.com/install.sh | sh

Manual install instructions

Docker

The official Ollama Docker image ollama/ollama is available on Docker Hub.

Libraries

Community

Get started

ollama

You'll be prompted to run a model or connect Ollama to your existing agents or applications such as Claude Code, OpenClaw, OpenCode , Codex, Copilot, and more.

Coding

To launch a specific integration:

ollama launch claude

Supported integrations include Claude Code, Codex, Copilot CLI, DeepSeek Harness, Droid, and OpenCode.

AI assistant

Use OpenClaw to turn Ollama into a personal AI assistant across WhatsApp, Telegram, Slack, Discord, and more:

ollama launch openclaw

Chat with a model

Run and chat with Gemma 4:

ollama run gemma4

See ollama.com/library for the full list.

See the quickstart guide for more details.

REST API

Ollama has a REST API for running and managing models.

curl http://localhost:11434/api/chat -d '{
  "model": "gemma4",
  "messages": [{
    "role": "user",
    "content": "Why is the sky blue?"
  }],
  "stream": false
}'

See the API documentation for all endpoints.

Python

pip install ollama

from ollama import chat

response = chat(model='gemma4', messages=[
  {
    'role': 'user',
    'content': 'Why is the sky blue?',
  },
])
print(response.message.content)

JavaScript

npm i ollama

import ollama from "ollama";

const response = await ollama.chat({
  model: "gemma4",
  messages: [{ role: "user", content: "Why is the sky blue?" }],
});
console.log(response.message.content);

Supported backends

  • llama.cpp project founded by Georgi Gerganov.

Documentation

Community Integrations

Want to add your project? Open a pull request.

Chat Interfaces

Web

Desktop

  • Dify.AI - LLM app development platform
  • AnythingLLM - All-in-one AI app for Mac, Windows, and Linux
  • Maid - Cross-platform mobile and desktop client
  • Witsy - AI desktop app for Mac, Windows, and Linux
  • Cherry Studio - Multi-provider desktop client
  • Ollama App - Multi-platform client for desktop and mobile
  • PyGPT - AI desktop assistant for Linux, Windows, and Mac
  • Alpaca - GTK4 client for Linux and macOS
  • SwiftChat - Cross-platform including iOS, Android, and Apple Vision Pro
  • Enchanted - Native macOS and iOS client
  • RWKV-Runner - Multi-model desktop runner
  • Ollama Grid Search - Evaluate and compare models
  • macai - macOS client for Ollama and ChatGPT
  • AI Studio - Multi-provider desktop IDE
  • Reins - Parameter tuning and reasoning model support
  • ConfiChat - Privacy-focused with optional encryption
  • LLocal.in - Electron desktop client
  • MindMac - AI chat client for Mac
  • Msty - Multi-model desktop client
  • BoltAI for Mac - AI chat client for Mac
  • IntelliBar - AI-powered assistant for macOS
  • Kerlig AI - AI writing assistant for macOS
  • Hillnote - Markdown-first AI workspace
  • Perfect Memory AI - Productivity AI personalized by screen and meeting history

Mobile

SwiftChat, Enchanted, Maid, Ollama App, Reins, and ConfiChat listed above also support mobile platforms.

Code Editors & Development

Libraries & SDKs

Frameworks & Agents

RAG & Knowledge Bases

  • RAGFlow - RAG engine based on deep document understanding
  • R2R - Open-source RAG engine
  • MaxKB - Ready-to-use RAG chatbot
  • Minima - On-premises or fully local RAG
  • Chipper - AI interface with Haystack RAG
  • ARGO - RAG and deep research on Mac/Windows/Linux
  • Archyve - RAG-enabling document library
  • Casibase - AI knowledge base with RAG and SSO
  • BrainSoup - Native client with RAG and multi-agent automation

Bots & Messaging

Terminal & CLI

Productivity & Apps

Observability & Monitoring

  • Opik - Debug, evaluate, and monitor LLM applications
  • OpenLIT - OpenTelemetry-native monitoring for Ollama and GPUs
  • Lunary - LLM observability with analytics and PII masking
  • Langfuse - Open source LLM observability
  • HoneyHive - AI observability and evaluation for agents
  • MLflow Tracing - Open source LLM observability

Database & Embeddings

Infrastructure & Deployment

Cloud

Package Managers

About

A fork of ollama/ollama adding vision-model correctness fixes — image token budgets, dynamic resolution, and an fp16-overflow gate for qwen2.5-vl — plus a vision regression suite. Tracks upstream releases; fixes are offered upstream and dropped here once they land.

ollama.com

Resources

Readme
License

Contributing

Contributing

Security policy

Security policy
Activity
Custom properties

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages