[seanpaulconnelly]

Kev-27B served through MLX on a 128 GB M5 Max: v2 (full weights, the path from #200) and v1 (the LoRA adapter). This PR adds the three reports and changes no docs or code.

v2 (full weights)

jaredpalmer/kev-27b @ af0e6d5 on kev main @ 84847f0, no code changes:

$ uv run --extra serve python -m kev.serve --run jaredpalmer/kev-27b --port 8009
serving jaredpalmer/kev-27b (...) on mps via mlx (bfloat16) 127.0.0.1:8009; states over 65,536 tokens refused (422)

Accuracy. kev.benchmark --remote on transfer-v4 development (clean subset, n = 656; 764 requested, 0 rejected). Committed as runs/mlx-27b-v2-transfer-v4/report.json. The rule from #200 (accuracy within ±0.015 of 0.851, Brier within ±0.02) holds: the accuracy gap is one question of 656.

	Published (runs/release/kev-27b-r23.json)	MLX bf16, served, M5 Max
Accuracy	0.8506	0.8491
Brier	0.2179	0.2182
ECE	0.0376	0.0344
Coverage at 5% error	0.6905	0.6753

Memory. footprint on the server process, sampled about once a second from process start through the benchmark:

	Bytes	GB	As footprint prints it (GiB)
Weights on disk (11 shards)	51,249,293,960	51.2	47.7
Steady state while serving (read after the benchmark)	53,650,511,384	53.7	50
Peak (phys_footprint_peak, whole run)	54,396,032,512	54.4	51

Right after load, before the first request, footprint printed 48 GB; it settled at 50 GB once requests ran, and the peak came during the benchmark, not at load. So the estimate in #200 holds: the peak is the weights plus about 3 GB, and there is no load spike. No sysctl change was needed. One unit note: footprint prints GiB, so v1's "52 GB steady, 97 GB peak" below are about 56 GB and 104 GB on the scale where the v2 weights are 51 GB.

Load and latency. The footprint reached its loaded level 6 seconds after process start, and the server was answering within about 10. The shards had just been downloaded, so they were probably still in the page cache; a cold load will take longer. Benchmark latency per request: median 360 ms, p95 428 ms.

v1 (LoRA adapter, tag v1-lora)

Measured on 2026-09-28 at kev @ 3e1cd3b, through the adapter-merge path:

• Accuracy: transfer-v4 development 0.8491 against the published 0.8476, Brier 0.2290 against 0.2289 (runs/mlx-27b-transfer-v4/report.json).
• Parity: scripts/mlx_parity.py --n 40 against the torch bf16 path: max |dp| 0.022, mean 0.0035, 0 argmax flips (runs/mlx-parity-27b/report.json). The script printed this report and then failed to write it because the directory didn't exist; the committed file is that printed JSON, not a rerun.
• Memory: 52 GB steady, 97 GB peak during the merge, as footprint printed them.

Test plan

• kev.benchmark --remote ... --suite evals/v4/transfer-v4 (development) against the v2 MLX server: 0.8491, n = 656 clean
• KEV_BASE_URL=http://127.0.0.1:8009 pytest tests/test_api.py against the v2 MLX server: 12 passed, 1 skipped
• uv run python scripts/verify_claims.py: verified 998 claims
• tests/test_conventions.py: 21 passed
• Setup: M5 Max, 128 GB, macOS 26.5; Python 3.13.13, mlx-lm 0.31.3, mlx 0.32.2, torch 2.8.0

[jaredpalmer]

Hi Sean, thank you for this! Getting Kev-27B running on a 128 GB M5 Max, and checking it against the published numbers, was really useful to us.

One update: Kev-27B v2, now jaredpalmer/kev-27b main, is no longer a LoRA adapter. It's a full-weight checkpoint, 51 GB of bf16 weights plus the pointer head, so the adapter-merge path your PR used doesn't apply to it. #200 adds a direct MLX path for full weights. It builds the model from the checkpoint's own config and shards and merges nothing. On Kev-4B written out as full weights, loading peaked at exactly the size of the weights (8.4 GB, against 15.9 GB for the merge), and answers were within 0.015 of the fp32 path. So v2 should need about 51 GB plus working memory, instead of the 97 GB peak you saw at v1's merge. We only have 32 GB Macs, though, so that's still an estimate.

If you have time, could you run v2 on your 128 GB Mac? Pull main and start uv run --extra serve python -m kev.serve --run jaredpalmer/kev-27b --port 8009 (it should pick MLX on its own). Then run uv run python -m kev.benchmark --remote http://127.0.0.1:8009 --suite evals/v4/transfer-v4 --out runs/mlx-27b-v2-transfer-v4 on the development split and note the memory footprint (footprint on the server process: steady state and peak). If accuracy lands within about ±0.015 of the published 0.851, we'd love to cite your numbers in the README and model card. And we'd be happy to fold your v1 report in alongside. Thanks again!

[seanpaulconnelly]

Thanks, Jared! Ran v2 on the 128 GB M5 Max, on main @ 84847f0 with jaredpalmer/kev-27b @ af0e6d5. It picked MLX on its own (mps via mlx (bfloat16)).

• transfer-v4 development: 0.8491 against the published 0.8506 (one question of 656), Brier 0.2182 against 0.2179. Inside your ±0.015 rule.
• Footprint: 53.7 GB steady while serving, 54.4 GB peak, for 51.2 GB of weights. The peak came during the benchmark, not at load, and no sysctl change was needed.
• Load: weights in memory 6 seconds after process start, with the shards freshly downloaded (so probably in the page cache). Benchmark latency: median 360 ms, p95 428 ms.

One unit note: footprint prints GiB, so it shows those two as 50 GB and 51 GB. My v1 numbers (52 GB steady, 97 GB peak) were read off the same display, so they are about 56 GB and 104 GB on the scale where the weights are 51 GB.

I rebuilt the PR on current main and dropped my doc edits, since main's wording replaced them. It now holds only three reports: runs/mlx-27b-v2-transfer-v4, plus v1's runs/mlx-27b-transfer-v4 and runs/mlx-parity-27b. I also updated the PR description to match; it has the full tables. Cite whatever is useful.