GPU passthrough for Ollama + embedding benchmark vs. the current CPU-bound setup

[JSONbored]

Part of #4325.

Context

src/review/rag.ts's embedding path currently runs against CPU-only Ollama inference, which has been measured to take considerably longer per batch than is comfortable under load, contributing to embedding timeouts and general resource contention with the rest of the review pipeline. The new host's GPU is provisioned specifically to fix this. nomic-embed-text (the current embedding model) is small — this is not a "does the GPU have enough VRAM" question, it's a "is passthrough actually wired correctly and does it measurably help" question.

Requirements

• NVIDIA Container Toolkit (or equivalent) installed on the new host so docker run --gpus all actually reaches the card — confirm with nvidia-smi both on the host and inside a test container before touching the real ollama service.
• docker-compose.yml's ollama service updated (locally, on the new host only) to request the GPU device.
• Pull the same model (nomic-embed-text) the current host runs; do not change models as part of this issue.
• Benchmark: same batch size, same repo's chunk set, GPU vs. the current CPU host's measured latency — record both, not just the new number.
• Re-validate the embed batch size setting (if in use) against the new hardware — a batch size tuned for CPU contention may be needlessly conservative on GPU; this issue should record a recommendation, not necessarily change the shipped default.

Deliverables

• nvidia-smi confirms GPU visibility inside a test container
• ollama service on the new host running model inference on GPU, confirmed via nvidia-smi showing utilization during an embed call (not falling back to CPU silently)
• Before/after latency numbers for the same embedding workload (CPU on the current host vs. GPU on the new host)
• A recorded recommendation for the embed batch size on the new hardware
• Compose changes for this stay local to the new host during the parallel-run window — no repo commit until cutover is decided (this issue is infrastructure config, not application code)

Expected outcome

Confirmed, measured, faster embedding throughput on the new host with no silent CPU fallback, and a data-backed batch-size recommendation instead of a guess.

Effort

M

[linear-code]

JSO-3303

[JSONbored]

GPU passthrough is confirmed working on the new host (nvidia-smi visible inside a test container, docker inspect gittensory-ollama-1 shows the nvidia device reservation via a local, uncommitted docker-compose.local-gpu.yml override — kept local per this issue's own "no repo commit until cutover is decided" instruction, and cutover has now happened).

Benchmark: 96-item batch (matching production EMBED_BATCH), real chunk text pulled from repo_chunks (avg ~6KB/chunk) via bge-m3:

• 19.7s total, ~205ms/chunk, GPU utilization confirmed active during the call (not a silent CPU fallback).

Historical CPU baseline (from migrated Loki logs, pre-cutover rag-index-repo job latencies — old host is now decommissioned so a live rerun isn't possible): median 3.9s/job, p75 15.9s/job, with occasional multi-minute outliers (max observed 60min) consistent with CPU contention from the rest of the pipeline. Not a perfectly clean per-item comparison since job batch sizes varied, but the GPU number shows no long-tail — consistent with removing the contention this issue set out to fix.

Batch-size recommendation: EMBED_BATCH is currently a hardcoded const = 96 in src/review/rag.ts with no env override (unlike its sibling RAG_DIMENSIONS/QDRANT_DIM). Filing a small follow-up PR to make it env-configurable via AI_EMBED_BATCH, mirroring the existing ragDimensionsFromEnv pattern, so this can be tuned for GPU throughput without a code change each time. No default change — 96 stays the shipped default for both hosted and self-host until real tuning data says otherwise.

Also worth noting: no dedicated Prometheus metric exists for embedding-call latency specifically (only generic gittensory_http_request_duration_seconds) — worth a small follow-up to instrument this properly for future benchmarking.

[JSONbored]

Re-benchmarked (2026-07-10) now that the follow-up tuning from this issue's own comment has landed on edge-nl-01: EMBED_BATCH env-configurable (#4350) and Ollama concurrency tuning (#4352), both merged to main and deployed as part of bringing the host current (see #4325's closing summary).

Benchmark: 96-item batch, real chunk text pulled fresh from repo_chunks (avg ~7.0KB/chunk, comparable to the original run) via bge-m3, direct timed call against the Ollama /v1/embeddings endpoint:

• 5.05s total, ~52.6ms/chunk — roughly 3.9x faster than this issue's original 19.7s / ~205ms-per-chunk result.
• GPU utilization confirmed peaking at 81% during the call (sampled every 1s throughout), VRAM ~947MiB — genuinely GPU-engaged, not a silent CPU fallback.

Not an apples-to-apples isolated A/B (the concurrency tuning and any incidental model/driver changes between the two runs aren't separated out), but a real, current, GPU-confirmed measurement showing the tuning work is paying off in practice.