local-llm-lab

Personal lab for evaluating local LLMs as coding-agent backends behind Pi on a 24 GB MacBook Pro M5 Pro.

The lab treats rapid-mlx and llama.cpp as equal first-class runners, with Unsloth Studio as an optional GUI lane that manages models and exposes the same kind of local OpenAI-compatible API. The goal is not "best MLX model" — it's best local coding-agent model that actually fits this machine.

Full design: docs/superpowers/specs/2026-05-21-model-stats-dashboard-design.md

Current model table: MODELS.md

What the lab does

Each model run uses Pi to build (or attempt to build) the same fullstack dashboard — a small app that tracks provenance, capabilities, and developer-feedback ratings for local models. The dashboard the local models build is also the place you record how each model did at building it.

Signals measured per model, in priority order:

1. Tool-use reliability — does the model + server combo drive Pi's file tools without going off the rails? Gate signal.
2. Completion depth — how far down the 8-phase ladder did it get?
3. Per-phase output quality — 4 ratings (1–5) recorded in the dashboard.

Hardware

• MacBook Pro M5 Pro, 24 GB unified memory.
• One ~17 GB-class model fits at a time. Close other heavy apps when running 27B-class models. Runs are sequential, not parallel.

Stack

Layer	Tool
Agent harness	pi (@earendil-works/pi-coding-agent)
Local server	rapid-mlx or llama.cpp on localhost:8000, or Unsloth Studio on localhost:8888 — all OpenAI-compatible
Model format	MLX 4-bit / mixed quant or GGUF
Dashboard frontend	Vite + React + TanStack Router + TanStack Query + Tailwind
Dashboard backend	Hono + Drizzle + better-sqlite3

Per-model lifecycle

# 1. Make a branch + worktree for this model
./scripts/new-run.sh qwen36-27b
#    → branch run/qwen36-27b, worktree at ./worktrees/qwen36-27b/

# 2. Terminal A — start the model (pick one runner)
# rapid-mlx example
rapid-mlx serve devstral-v2-24b

# llama.cpp example
llama-server -hf <user>/<model>:<quant> --host 127.0.0.1 --port 8000 -np 1 -c 4096

# 3. Terminal B — drive Pi inside the worktree's skeleton/ dir
cd worktrees/qwen36-27b/skeleton
pi --no-skills
# or, when project-local benchmark skills exist under skeleton/.pi/skills:
# pi --no-skills \
#   --skill .pi/skills/<skill-a>/SKILL.md \
#   --skill .pi/skills/<skill-b>/SKILL.md
#    inside pi:  /model  → pick the running model
#    opening prompt:
#    > Read PLAN.md and work through it phase by phase.
#    > After each phase, stop and wait for me to confirm before continuing.

# 4. After each phase: observe, rate, commit with phase tag
#    (from the worktree root)
#    git commit -am "phase 3: seed"

# 5. End of run: copy and fill out the template (on the main checkout)
cp runs/_TEMPLATE.md runs/qwen36-27b.md

Compare runs across models with: git diff run/qwen36-27b..run/llama32-3b -- skeleton/

Recommended Pi setup for local-model benchmark runs

For benchmark consistency, the recommended default is to run Pi from inside worktree/skeleton/ with global skill discovery disabled.

Why:

• benchmark runs should not depend on whatever global skills happen to be installed on your machine
• small local models are easier to evaluate when the tool/skill environment is narrow and predictable
• this keeps failures attributable to the model, prompt, runner, or repo — not to unrelated personal global agent customisation

Recommended command shape:

cd worktrees/<model-slug>/skeleton
pi --no-skills

If you add benchmark-specific local skills under skeleton/.pi/skills/, load only the ones you want explicitly:

cd worktrees/<model-slug>/skeleton
pi --no-skills \
  --skill .pi/skills/<skill-a>/SKILL.md \
  --skill .pi/skills/<skill-b>/SKILL.md

Important Pi behavior: --no-skills disables all skill discovery, including project-local .pi/skills/. So the benchmark-safe workflow is: store local benchmark skills in skeleton/.pi/skills/, but pass them explicitly with repeatable --skill flags.

If no local benchmark skills exist yet, still prefer pi --no-skills for the baseline run.

Local benchmark skills worth adding later

Early runs suggest a small local benchmark skill pack would help local models avoid common failures:

1. Latest package docs fetcher
   • quickly fetch current docs/examples for fast-moving packages
   • especially useful for drizzle-orm, where outdated schema syntax or type assumptions can break migration work
2. Architecture self-check
   • force the model to restate the current app shape before coding
   • especially useful for SPA frontend + Hono API + Drizzle SQLite structure
3. Migration/schema sanity check
   • before running migrations, verify schema imports, Drizzle syntax, and expected generated files against the installed package version

These should stay local to the benchmark repo so every tested model gets the same narrow support surface.

The phase ladder (lives in skeleton/PLAN.md)

#	Phase	Deliverable
1	Read & ack	Summarize project; list npm scripts; list files for phase 2. No code.
2	Schema	src/db/schema.ts with models + feedback tables; Drizzle migration
3	Seed	src/db/seed.ts inserts 3 fixture rows; wire and run db:seed
4	API: list	GET /api/models Hono route, typed
5	List view	/models sortable table via TanStack Query
6	Detail view	/models/$id route + GET /api/models/:id
7	Feedback form	4 ratings 1–5 + notes; POST /api/models/:id/feedback; cache invalidation
8	Ratings rollup	"Avg" column on list view; sort by avg desc

Phase 7 is the canary — most small models break here. Phase 1 produces no code on purpose: it's the cheapest possible tool-use gate.

Scoring per phase

Recorded in runs/<model-slug>.md (copy of runs/_TEMPLATE.md):

Outcome	Meaning
✅ pass	Works on first try; may be inelegant
🟡 pass-with-nudge	One short correction needed from user
❌ fail	Stuck in a loop, broken code, hallucinated APIs, or gave up — run ends
⏭ skipped	Run ended earlier; phase never attempted

Aggregate per model = max phase reached + nudge count + the four 1–5 ratings.

Canonical conclusions live in runs/_TEMPLATE.md and the per-model files copied from it under runs/. README tracks the benchmark setup; runs/*.md tracks what actually happened.

Install (once, already done)

Pi

curl -fsSL https://pi.dev/install.sh | sh

rapid-mlx

brew install raullenchai/rapid-mlx/rapid-mlx
rapid-mlx --version
rapid-mlx models

llama.cpp

brew install llama.cpp
llama-server --version

Unsloth Studio

Installed via the unsloth CLI (lives under ~/.unsloth):

unsloth --version
unsloth studio            # web UI + OpenAI-compatible API on :8888

An unsloth-local wrapper (in ~/.local/bin) launches Studio with HF_HUB_OFFLINE=1 so it loads already-downloaded models without re-downloading — see the Unsloth Studio runner notes above for why.

GGUF downloads helper

python3 -m pip install -U "huggingface_hub[cli]"
hf --help

Note: recent huggingface_hub releases expose the CLI as hf rather than huggingface-cli.

Optional: Hugging Face auth for faster downloads

Unauthenticated downloads work, but Hugging Face may apply lower rate limits. For better reliability and speed:

1. Create a read token at https://huggingface.co/settings/tokens
2. Log in with the CLI:

hf auth login

Or set an environment variable manually:

export HF_TOKEN="hf_xxx_your_token_here"

To persist it in zsh:

echo 'export HF_TOKEN="hf_xxx_your_token_here"' >> ~/.zshrc
source ~/.zshrc

Verify auth with:

hf auth whoami

Currently installed: Pi (npm global), rapid-mlx 0.6.61, Unsloth Studio (unsloth CLI). The lab supports rapid-mlx and llama.cpp runner lanes plus the Unsloth Studio GUI lane.

Model candidates

The current candidate table lives in MODELS.md. Keep fast-moving model choices, tried/not-tried status, memory notes, and suggested startup commands there. Keep full benchmark results in runs/<model-slug>.md.

Practical rule: do not start with the 24B–30B models. Use smaller models to validate Pi, prompt wording, runner config, and your scoring rhythm before burning time on tight-memory candidates.

Runner startup commands

Use the exact candidate-specific startup commands in MODELS.md. These examples show the two runner shapes only.

rapid-mlx

rapid-mlx serve <alias-or-hf-repo> \
  --host 127.0.0.1 \
  --port 8000

Conservative flags for 24 GB runs:

rapid-mlx serve devstral-v2-24b \
  --host 127.0.0.1 \
  --port 8000 \
  --max-num-seqs 1 \
  --max-concurrent-requests 4 \
  --cache-memory-percent 0.10 \
  --gpu-memory-utilization 0.85

If a larger model is unstable, lower memory pressure first:

• close browsers / Docker / IDE extras
• keep --max-num-seqs 1
• reduce cache budget
• only then decide the model is too large

llama.cpp

llama-server \
  -hf <user>/<model>:<quant> \
  --host 127.0.0.1 \
  --port 8000 \
  -np 1 \
  -c 4096

For multimodal GGUFs, keep mmproj.gguf next to the main model and pass the appropriate llama.cpp vision flags when needed. The first-pass benchmark remains text/tool-calling focused.

Unsloth Studio (GUI lane)

Unsloth Studio is an alternative model-management GUI. You download and load models in a browser, and it serves an OpenAI-compatible API a coding agent can talk to. It bundles and auto-manages its own llama.cpp/MLX backend, so you do not start a server yourself.

# Launch the web UI + API on http://127.0.0.1:8888
unsloth studio

Connect an agent to the model currently loaded in Studio:

unsloth connect opencode   # or: codex | claude | openclaw | hermes

Pi isn't a built-in connect target, so point Pi at Studio's OpenAI-compatible endpoint on :8888 using the API key Studio mints per server (env var UNSLOTH_API_KEY). Note Studio uses :8888, not the :8000 the CLI runners use.

Gotcha — loading an already-downloaded model. Studio tracks each repo's HuggingFace main. When Unsloth re-uploads a GGUF its bytes change, so on the next load Studio re-downloads the new revision even though a complete older copy is in your cache — and it mislabels that multi-GB download as "Loading cached model into memory." To force Studio to use the local copy, launch it offline:

HF_HUB_OFFLINE=1 HF_HUB_DISABLE_XET=1 unsloth studio   # saved as the `unsloth-local` launcher

Offline mode also stops Studio from rewriting the cache pointer (refs/main) back to the upstream revision, which is what otherwise undoes the fix. Launch plain unsloth studio only when you actually want to download or upgrade a model. If a previously-downloaded model still won't resolve offline, its cache refs/main may point at an empty re-uploaded snapshot — re-download it once online, or repoint the ref to the revision that holds the file.

Repo layout

local-llm-lab/
├── README.md                     # this file
├── MODELS.md                     # candidate table and manual run status
├── docs/superpowers/specs/       # design docs
├── skeleton/                     # pre-scaffolded fixture (PLAN.md lives here)
├── worktrees/                    # per-run git worktrees created by new-run.sh
├── runs/
│   ├── _TEMPLATE.md              # copy per model run
│   └── <model-slug>.md           # one per run, committed
└── scripts/new-run.sh            # branch + worktree + copy skeleton