chore: bench_decode.sh all measures duplicate checkpoints twice #1615

Body rewritten 2026-09-04. The body originally filed here was a copy of #1616 (batched MoE decode scaling) and did not describe this issue at all; the two differed only in cosmetic wording. Everything below is reconstructed from the script and from the committed benchmark CSVs. The title was correct.

Problem / Background

bench_decode.sh all enumerates "$MODELS_DIR"/*/ and benchmarks every directory it finds (scripts/bench_decode.sh:658 and :666). Nothing deduplicates, so a checkpoint present under two directory names is loaded, prefilled, decoded and cooled down twice, and lands in the CSV as two rows under two names. Downstream the duplicate rows are treated as two distinct models.

Evidence

Scanning models/mlx on M1 Ultra by sha256(config.json) plus total *.safetensors bytes finds 15 duplicate groups covering 16 redundant directories out of 199, holding 124.5 GB of redundant weights.

Six groups were additionally spot-checked by shard count plus a sha256 over the first and last 1 MiB of every shard: all six match, and none share inodes, so these are independent full copies rather than hardlinks or APFS clones. Names are misleading in both directions: pixtral-12b and pixtral-12b-4bit are the same 4-bit checkpoint, and hunyuan-13b is the same file set as hunyuan-a13b-instruct-4bit.

The duplication is in every committed sweep, not just on this host. Counting group members that co-occur in each committed CSV:

The paired numbers agree to within run noise, which is itself confirmation that the two rows measure one checkpoint: qwen2.5-7b / qwen2.5-7b-4bit / qwen2.5-7b-instruct-4bit decode at 108.85 / 108.47 / 108.86 tok/s in metal_m1ultra_2026-07-12.csv, and nemotron-h-30b-4bit / nemotron-nas-30b-4bit at 91.75 / 90.37. The VLM sweeps carry the same redundancy at the same counts.

It reaches published documentation. Three duplicate groups appear as separate table rows in each of docs/benchmark_results/model_tests_m1ultra.md and docs/benchmark_results/model_tests_m5max.md (gemma-3-4b-it-4bit / gemma3-4b-4bit, nemotron-h-30b-4bit / nemotron-nas-30b-4bit, pixtral-12b / pixtral-12b-4bit, qwen3-30b-a3b-4bit / qwen3-moe-4bit depending on the file), so one checkpoint is presented as two supported models. The parity comparison tables in the same documents list only one member per group, which means the deduplication currently exists only as unrecorded manual curation between the CSV and the document. That is not reproducible and it silently drifts every time someone regenerates a table.

Current Behavior

if [[ "$MODEL_ARG" == "all" ]]; then
  # First pass: run all models except known GPU-crash models
  for dir in "$MODELS_DIR"/*/; do
    [[ -d "$dir" ]] || continue
    is_gpu_crash_model "$dir" && continue
    result=$(bench_one "$dir")

The two-pass split at :658 and :666 exists to defer GPU_CRASH_MODELS (:603) to the end; each directory is still visited exactly once per sweep. The defect is that two directories holding one checkpoint are two directories. bench_one (:414) keys everything on the directory basename, and emit (:595) appends whatever it returns, so the CSV inherits the directory naming with no identity notion at all.

Proposed Solution

Deduplicate by checkpoint identity in all mode, before the first pass, and keep single-model mode untouched so an explicit path always measures exactly what was named.

• Compute a cheap identity key per candidate directory: sha256(config.json) plus the sorted list of (*.safetensors basename, size). This costs one small read plus a stat per shard, needs no weight hashing, and correctly separates checkpoints that differ only in quantization (their config.json differs in the quantization block).
• Also collapse directories that resolve to the same path, so a symlink such as models/mlx/gemma-3-1b-it-4bit into the shared model store never doubles a row.
• Measure one member per group and skip the rest, emitting one row per skipped alias with a SKIP:duplicate_of=<name> status so the CSV stays self-describing and a reader can see the alias set rather than silently wondering where a model went. Row count then stays stable across sweeps, which matters because docs/benchmark_results/model_tests.md indexes sweeps by measured row count. The survivor only has to be deterministic; as implemented it is the first directory the sweep's own glob reaches, which means the longer name wins for a prefix pair (pixtral-12b-4bit over pixtral-12b).
• Print the collapsed groups to stderr at sweep start so the operator sees what was folded before a multi-hour run begins.
• Add --no-dedup to restore today's behavior for anyone who deliberately wants to measure two copies, for example when checking whether a re-download is bit-identical.

Rejected: deleting the redundant directories. models/ is gitignored local state, the aliases are how various scripts and documents already refer to these checkpoints (scripts/bench_embeddings.py, the deferred-pairing tables), and a sweep must not be silently wrong on a host that happens to hold two copies. The harness has to be correct regardless of how the model store is arranged.

Scope

In scope: scripts/bench_decode.sh (all mode enumeration, the new dedup helper, --no-dedup, usage text at :285), and a note in docs/benchmarks.md describing the identity key and the SKIP:duplicate_of status.

Out of scope: deleting or renaming any directory under models/; regenerating past sweeps; correcting the duplicate rows already present in committed CSVs and in model_tests_m1ultra.md and model_tests_m5max.md, which is a documentation pass that should follow once sweeps stop producing them; scripts/bench_embeddings.py and scripts/bench_serving_concurrency.py, which take explicit model lists rather than enumerating a directory.

Acceptance Criteria

• bench_decode.sh all measures each distinct checkpoint exactly once, with the alias names emitted as SKIP:duplicate_of=<name> rows so total row count is unchanged.
• The identity key does not collapse checkpoints that differ only in quantization or in weight dtype; a test or a documented check demonstrates this on a real pair from models/mlx, for example bitnet-b1.58-2b-4t against bitnet-b1.58-2b-4t-4bit.
• A directory that is a symlink to another measured directory is collapsed as well.
• --no-dedup reproduces the current behavior exactly.
• Single-model mode (bench_decode.sh models/<name>) is unaffected and still measures the named path.
• The collapsed groups are printed to stderr before the first measurement.
• docs/benchmarks.md documents the identity key and the new status string.

Verification

# Sanity-check the script and see the grouping the sweep would apply. The
# collapsed groups print to stderr before the first model is measured, so a
# sweep against a small temporary MODELS_DIR shows them without a long run.
bash -n scripts/bench_decode.sh

# Cheap end-to-end check: build a temporary MODELS_DIR of symlinks holding two
# small duplicate groups (gemma-3-1b-it-4bit / gemma3-1b-4bit and qwen2-0.5b /
# qwen2.5-0.5b-4bit) plus one non-duplicate control, then run both arms.
MODELS_DIR=<tmpdir> ./scripts/bench_decode.sh all --output /tmp/dedup.csv
MODELS_DIR=<tmpdir> ./scripts/bench_decode.sh all --no-dedup --output /tmp/nodedup.csv

A pass is: one measured row per distinct checkpoint, the alias rows present and labelled, the sweep's total row count equal to the directory count so the CSV index in docs/benchmark_results/model_tests.md stays meaningful, and --no-dedup measuring every directory. There is no --dry-run; an earlier draft of this body referenced one that was never implemented, and the stderr group report plus a temporary MODELS_DIR cover the same need.

Technical Considerations

Relevant to #1614, which compares per-checkpoint prefill across two sweeps: duplicate rows make a "checkpoint" ambiguous in exactly the comparison that issue performs. Filed alongside #1612, #1613, #1614 and #1616 from the same 2026-09-04 benchmark campaign.