Sudoku Silver — MLX Fine-Tuning on Apple Silicon

Fine-tune an LLM to solve Sudoku puzzles using Apple's MLX framework with LoRA. No GPU required — Apple Silicon unified memory handles everything.

---

Quick Start

# 1. Install uv (if needed)
curl -LsSf https://astral.sh/uv/install.sh | sh

# 2. Install dependencies
uv sync

# 3. Generate dataset (2000 puzzles with train/valid/test splits)
uv run sudoku-generate

# 4. Evaluate baseline (BEFORE fine-tuning — how good is the base model?)
uv run sudoku-eval --mode baseline

# 5. Fine-tune (LoRA, ~20-30 min on M4 Pro)
uv run sudoku-train

# 6. Evaluate + compare (AFTER fine-tuning — did it improve?)
uv run sudoku-eval --mode compare

# 7. Solve puzzles interactively
uv run sudoku-solve

---

Project Structure

sukodu/
├── pyproject.toml                # uv-managed project + dependencies
├── README.md
├── sudoku_silver/                # main package
│   ├── __init__.py
│   ├── engine.py                 # Sudoku generation, solving, validation
│   ├── config.py                 # shared constants and paths
│   ├── dataset.py                # dataset generator (train/valid/test splits)
│   ├── finetune.py               # MLX LoRA fine-tuning launcher
│   ├── evaluate.py               # systematic pre/post evaluation pipeline
│   └── solve.py                  # interactive solver CLI
├── eval/
│   └── test_puzzles.json         # curated test puzzles with solutions
├── configs/
│   └── default.yaml              # training/eval configuration
├── data/                         # generated datasets
│   ├── train.jsonl               # training data
│   ├── valid.jsonl               # validation data
│   └── test.jsonl                # held-out test set (never seen during training)
├── models/                       # model artifacts
│   ├── adapters/                 # LoRA adapter weights
│   └── fused/                    # fused model (base + adapter)
├── results/                      # evaluation reports
│   ├── baseline/                 # pre-fine-tuning results
│   │   └── eval_report.json
│   ├── finetuned/                # post-fine-tuning results
│   │   └── eval_report.json
│   └── comparison.json           # side-by-side delta report
├── tests/                        # pytest test suite
│   ├── test_engine.py
│   └── test_evaluate.py
└── outputs/                      # training logs and checkpoints

---

Evaluation Workflow (BEFORE vs AFTER)

The evaluation pipeline (uv run sudoku-eval) supports three modes:

Mode	What it does
--mode baseline	Tests the untrained base model on 6 curated puzzles. Saves to results/baseline/.
--mode finetuned	Fuses adapters, tests the fine-tuned model on the same puzzles. Saves to results/finetuned/.
--mode compare	Runs both and generates a comparison with deltas.

Metrics Tracked

• Board validity — does the output obey Sudoku rules? (rows/cols/boxes all have 1-9)
• Exact match — does it match the ground-truth solution exactly?
• Cell accuracy — what fraction of individual cells are correct?
• Difficulty breakdown — performance split by easy/medium/hard/expert
• Inference time — seconds per puzzle

Compare Output Example

  BEFORE vs AFTER Fine-Tuning Comparison
  Baseline : mlx-community/Qwen2.5-7B-Instruct-4bit
  Finetuned: ./models/fused/sudoku-silver

  Valid Rate          : 16.67% → 83.33%  (↑ +66.67%)
  Exact Rate          : 0.00% → 66.67%   (↑ +66.67%)
  Avg Cell Accuracy   : 28.50% → 91.20%  (↑ +62.70%)

  Per-Difficulty:
    easy     exact: 0.0% → 100.0%  (↑ +100.0%)
    medium   exact: 0.0% → 66.7%   (↑ +66.7%)
    hard     exact: 0.0% → 50.0%   (↑ +50.0%)
    expert   exact: 0.0% → 33.3%   (↑ +33.3%)

---

Customization

Generate more/fewer puzzles

uv run sudoku-generate --count 5000        # more data
uv run sudoku-generate --count 1000        # faster
uv run sudoku-generate --test 0.15         # larger held-out test set

Choose a different base model

# Fastest (recommended)
uv run sudoku-train --model mlx-community/Qwen2.5-7B-Instruct-4bit

# Full precision
uv run sudoku-train --model Qwen/Qwen2.5-7B-Instruct

# Lightweight
uv run sudoku-train --model microsoft/Phi-3-mini-4k-instruct

# Use shorthand keys
uv run sudoku-train --model qwen7b_4bit

Train longer for better results

uv run sudoku-train --iters 2000 --lora-layers 32

Memory-constrained

uv run sudoku-train --batch-size 2 --lora-layers 8 --grad-checkpoint

Evaluate a specific model

uv run sudoku-eval --mode baseline --model mlx-community/Mistral-7B-Instruct-v0.2-4bit

---

Training Time Estimates (M4 Pro 24GB)

Model	Iters	Approx Time
Qwen2.5-7B-Instruct-4bit	1000	~15-20 min
Qwen2.5-7B-Instruct (fp16)	1000	~25-30 min
Phi-3-mini-4k-instruct	1000	~10-12 min
Mistral-7B-Instruct-4bit	1000	~15-20 min

---

Running Tests

uv run pytest tests/ -v

---

Requirements

• macOS 14+ (Sonoma or Sequoia)
• Apple Silicon (M1/M2/M3/M4)
• Python 3.10+
• uv package manager
• Internet (first run downloads model from HuggingFace, ~4-14GB)