perf(bench): long-prompt prefill benchmark coverage and serving TTFT/decode-rate telemetry #624

Part of #623.

Context

All existing benchmark CSVs (benchmarks/cuda_gb10_2026-06-17.csv and the Metal sets) use prompts of 8-66 tokens. At that length prefill numbers are dominated by graph-build and kernel-launch overhead, not by matmul throughput, so they cannot validate any prefill optimization (MoE prefill fixes, sm_121 GEMM work, chunked prefill tuning). Separately, the server exposes only aggregate counters (src/server/batch/observability.rs); there is no TTFT or decode-rate telemetry, so serving-level changes (batching defaults, pipelining) cannot be measured except by external tooling.

This issue is the measurement foundation for the rest of the epic and must land first.

Scope

1. Long-prompt prefill benchmarking in the existing harness.
2. TTFT and decode-rate metrics in the server's Prometheus surface.
3. A concurrency load-generation script for serving-throughput measurements.

Implementation plan

1. Long-prompt prefill benches

• Extend src/bin/bench_decode.rs (binary mlxcel-bench-decode) and scripts/bench_decode.sh to accept a target prompt token length (e.g. --prompt-tokens N). Synthesize the prompt deterministically (repeat a fixed corpus paragraph, then truncate to N tokens with the model's tokenizer) so runs are reproducible across models.
• Standard lengths: 512, 2048, 8192, 32768 (cap at the model's max context; record the actual length used).
• Emit the same CSV schema as today (prefill_ms, prefill_tok_s, decode_ms, decode_tok_s) plus a prompt_target_len column. Keep the short-prompt default intact so historical CSV comparability is preserved.
• Add a wrapper mode in scripts/bench_all_models.sh (or a sibling script) that runs a representative subset (llama-3.1-8b-4bit, qwen2.5-7b-4bit, qwen3-30b-a3b-4bit, mixtral-8x7b-4bit, gemma-4-31b-it-4bit) across the length ladder, since a full 147-model sweep at 32k is too slow.

2. Server TTFT / decode-rate telemetry

• The worker already captures first_token_time per request (src/server/model_worker.rs:1937-1939) and computes prompt_eval_ms at finish (src/server/model_worker.rs:2110-2132). Surface these into BatchObservability (src/server/batch/observability.rs): add a TTFT histogram (or fixed-bucket summary) and a per-request decode tok/s histogram, recorded in finalize_completed (src/server/batch/scheduler.rs:4964 area) where the finished sequence's timings are available.
• Export both through /metrics (src/server/routes/metrics.rs, existing counter block around lines 144-155) with names like mlxcel_request_ttft_ms and mlxcel_request_decode_tok_s.
• Keep overhead negligible: record once per request completion, never per token.

3. Concurrency load script

• Add scripts/bench_serving_concurrency.py: an asyncio client using the OpenAI-compatible API that fires N concurrent streaming chat requests (configurable prompt length and max_tokens), and reports per-request TTFT, decode tok/s, and aggregate tok/s for N in {1, 2, 4, 8}.

Acceptance criteria

• mlxcel-bench-decode --prompt-tokens 8192 runs against a local checkpoint and emits a CSV row with the new column; short-prompt invocations produce byte-compatible rows with the old schema plus the new column.
• A long-prompt baseline CSV for the representative subset is committed under benchmarks/ (CUDA machine; name it cuda_gb10_longprompt_<date>.csv) together with a short note in docs/benchmark_results/ describing the ladder.
• /metrics exposes TTFT and decode-rate metrics populated by real requests (verify by curling /metrics after a few requests through mlxcel-server).
• scripts/bench_serving_concurrency.py produces a summary table against a running server.
• cargo test --features cuda passes locally; cargo clippy clean on touched crates.

Validation

cargo build --release --features cuda
./target/release/mlxcel-bench-decode --model ./models/llama-3.1-8b-4bit --prompt-tokens 8192
./target/release/mlxcel-server --model ./models/llama-3.1-8b-4bit &
python3 scripts/bench_serving_concurrency.py --concurrency 4
curl -s localhost:8080/metrics | grep -E "ttft|decode_tok"

References

• Benchmark harness docs: Cargo.toml [[bin]] mlxcel-bench-decode comment block; scripts/bench_decode.sh.
• Existing metrics: src/server/routes/metrics.rs:144-155, src/server/batch/observability.rs:133-184.
• Per-request timing fields: src/server/model_provider.rs:71-89.