spec : add DFlash2 support (local convolution + candidate selector) (#27342) by ngxson · Pull Request #27816 · ggml-org/llama.cpp

ngxson · 2026-08-27T17:08:35Z

I cannot push to #27342 because the PR was created from an org, applying @ORippler 's request to revert top-k.cu here

for full changes, please refer to #27342

…27342) * support DFlash2 * Add p_min in DFlash2 Assisted-by: Claude Opus 5 * Revert unnecessary changes Assisted-by: Claude Opus 5 * Revert draft sampling in rejection sampling Assisted-by: Claude Opus 5 * Refactor code structure Assisted-by: Claude Opus 5 * Delete embedding scaling Assisted-by: Claude Opus 5 * Gate output transforms on DFlash2 Assisted-by: Claude Opus 5 * Optimize Dflash 2 cost Assisted-by: Claude Opus 5 * Avoid using atoi Assisted-by: Claude Opus 5 * Modify comments Assisted-by: Claude Opus 5 * Move llama_model_dflash_selector_top_k to llama-ext.h Assisted-by: Claude Opus 5 * Formatting Assisted-by: Claude Opus 5 * Apply patch to fix the mrope bug Assisted-by: Claude Opus 5 * fix ci Assisted-by: Claude Opus 5 * Fix graph number calculation Assisted-by: Claude Opus 5 * rename hid and unary Assisted-by: Claude Opus 5 --------- Co-authored-by: Jian Chen <jianchen0311@gmail.com> Co-authored-by: Xuan-Son Nguyen <son@huggingface.co>

…gml-org#27342) (ggml-org#27816) * spec : add DFlash2 support (local convolution + candidate selector) (ggml-org#27342) * support DFlash2 * Add p_min in DFlash2 Assisted-by: Claude Opus 5 * Revert unnecessary changes Assisted-by: Claude Opus 5 * Revert draft sampling in rejection sampling Assisted-by: Claude Opus 5 * Refactor code structure Assisted-by: Claude Opus 5 * Delete embedding scaling Assisted-by: Claude Opus 5 * Gate output transforms on DFlash2 Assisted-by: Claude Opus 5 * Optimize Dflash 2 cost Assisted-by: Claude Opus 5 * Avoid using atoi Assisted-by: Claude Opus 5 * Modify comments Assisted-by: Claude Opus 5 * Move llama_model_dflash_selector_top_k to llama-ext.h Assisted-by: Claude Opus 5 * Formatting Assisted-by: Claude Opus 5 * Apply patch to fix the mrope bug Assisted-by: Claude Opus 5 * fix ci Assisted-by: Claude Opus 5 * Fix graph number calculation Assisted-by: Claude Opus 5 * rename hid and unary Assisted-by: Claude Opus 5 --------- Co-authored-by: Jian Chen <jianchen0311@gmail.com> Co-authored-by: Xuan-Son Nguyen <son@huggingface.co> * revert top-k.cu changes --------- Co-authored-by: Zihan Zhang <tiancaizhangdaxian@sjtu.edu.cn> Co-authored-by: Jian Chen <jianchen0311@gmail.com>

…gml-org#27342) (ggml-org#27816) * spec : add DFlash2 support (local convolution + candidate selector) (ggml-org#27342) * support DFlash2 * Add p_min in DFlash2 Assisted-by: Claude Opus 5 * Revert unnecessary changes Assisted-by: Claude Opus 5 * Revert draft sampling in rejection sampling Assisted-by: Claude Opus 5 * Refactor code structure Assisted-by: Claude Opus 5 * Delete embedding scaling Assisted-by: Claude Opus 5 * Gate output transforms on DFlash2 Assisted-by: Claude Opus 5 * Optimize Dflash 2 cost Assisted-by: Claude Opus 5 * Avoid using atoi Assisted-by: Claude Opus 5 * Modify comments Assisted-by: Claude Opus 5 * Move llama_model_dflash_selector_top_k to llama-ext.h Assisted-by: Claude Opus 5 * Formatting Assisted-by: Claude Opus 5 * Apply patch to fix the mrope bug Assisted-by: Claude Opus 5 * fix ci Assisted-by: Claude Opus 5 * Fix graph number calculation Assisted-by: Claude Opus 5 * rename hid and unary Assisted-by: Claude Opus 5 --------- Co-authored-by: Jian Chen <jianchen0311@gmail.com> Co-authored-by: Xuan-Son Nguyen <son@huggingface.co> * revert top-k.cu changes --------- Co-authored-by: Zihan Zhang <tiancaizhangdaxian@sjtu.edu.cn> Co-authored-by: Jian Chen <jianchen0311@gmail.com> (cherry picked from commit b10f9ca)

anfedoro · 2026-09-08T07:34:11Z

senseless until split-mode = tensor is stable.

…gml-org#27342) (ggml-org#27816) * spec : add DFlash2 support (local convolution + candidate selector) (ggml-org#27342) * support DFlash2 * Add p_min in DFlash2 Assisted-by: Claude Opus 5 * Revert unnecessary changes Assisted-by: Claude Opus 5 * Revert draft sampling in rejection sampling Assisted-by: Claude Opus 5 * Refactor code structure Assisted-by: Claude Opus 5 * Delete embedding scaling Assisted-by: Claude Opus 5 * Gate output transforms on DFlash2 Assisted-by: Claude Opus 5 * Optimize Dflash 2 cost Assisted-by: Claude Opus 5 * Avoid using atoi Assisted-by: Claude Opus 5 * Modify comments Assisted-by: Claude Opus 5 * Move llama_model_dflash_selector_top_k to llama-ext.h Assisted-by: Claude Opus 5 * Formatting Assisted-by: Claude Opus 5 * Apply patch to fix the mrope bug Assisted-by: Claude Opus 5 * fix ci Assisted-by: Claude Opus 5 * Fix graph number calculation Assisted-by: Claude Opus 5 * rename hid and unary Assisted-by: Claude Opus 5 --------- Co-authored-by: Jian Chen <jianchen0311@gmail.com> Co-authored-by: Xuan-Son Nguyen <son@huggingface.co> * revert top-k.cu changes --------- Co-authored-by: Zihan Zhang <tiancaizhangdaxian@sjtu.edu.cn> Co-authored-by: Jian Chen <jianchen0311@gmail.com>

…gml-org#27342) (ggml-org#27816) (#261) * spec : add DFlash2 support (local convolution + candidate selector) (ggml-org#27342) * support DFlash2 * Add p_min in DFlash2 Assisted-by: Claude Opus 5 * Revert unnecessary changes Assisted-by: Claude Opus 5 * Revert draft sampling in rejection sampling Assisted-by: Claude Opus 5 * Refactor code structure Assisted-by: Claude Opus 5 * Delete embedding scaling Assisted-by: Claude Opus 5 * Gate output transforms on DFlash2 Assisted-by: Claude Opus 5 * Optimize Dflash 2 cost Assisted-by: Claude Opus 5 * Avoid using atoi Assisted-by: Claude Opus 5 * Modify comments Assisted-by: Claude Opus 5 * Move llama_model_dflash_selector_top_k to llama-ext.h Assisted-by: Claude Opus 5 * Formatting Assisted-by: Claude Opus 5 * Apply patch to fix the mrope bug Assisted-by: Claude Opus 5 * fix ci Assisted-by: Claude Opus 5 * Fix graph number calculation Assisted-by: Claude Opus 5 * rename hid and unary Assisted-by: Claude Opus 5 --------- * revert top-k.cu changes --------- (cherry picked from commit b10f9ca) Co-authored-by: Xuan-Son Nguyen <son@huggingface.co> Co-authored-by: Zihan Zhang <tiancaizhangdaxian@sjtu.edu.cn> Co-authored-by: Jian Chen <jianchen0311@gmail.com>

DFlash 2 is merged upstream (ggml-org/llama.cpp#27816), so CI now builds llama-server from ggml-org/llama.cpp at b11213 instead of the z-lab fork. That picks up the fused DFlash encoder/KV injection (#27310), which removes the prompt-processing slowdown the fork had with speculation on (M1 Max, 4.4k-token prompt: 109 tok/s with DFlash vs 88 without; the fork was ~2x slower with DFlash). start.sh now runs a single slot (-np 1) with a RAM prompt cache, so an agent's long prompt is reused between turns instead of re-read, and sizes context to the machine's RAM. QUANT=Q8_0 selects the 8-bit target and draft. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

… for large rows (#2) Ports upstream DFlash2 support (b10f9ca, "spec : add DFlash2 support (local convolution + candidate selector)") onto this fork's DFlash code: - dflash.cpp: DFlash2 hparams (block/conv/selector keys), per-layer local convolutions around attention and FFN, the target's logit scale/softcap on the draft logits, and the selector lattice (top-k candidates + pairwise transition scores per block position) written to the pre-norm output (this fork's name for upstream's h_nextn) - speculative.cpp: the greedy lattice walk (p_min on the softmax of the winning score, n_min); block tokens request no logits for DFlash2 - llama-context: DFlash2 graph node budget - ggml-cuda: upstream's HIP radix top-k, so TOP_K over a 248k vocab stays on the GPU instead of falling back to the CPU (test-backend-ops TOP_K 445/445 on gfx1151, rows up to 524288) - common: keep bounded recurrent-state rollback (n_rs_seq) for draft-dflash, as upstream does; without it every partial accept restores a checkpoint and replays the accepted tokens M-RoPE handling from the upstream commit is not needed here: the Qwen3.8 drafter has one 64-wide section, identical to NEOX with its positions. Qwen3.8-27B (Hadamard-rotated Q4_0 + W4A4 prompt path) + Qwen3.8-27B-DFlash2 q8_0 on gfx1151, -ctxcp 0, --spec-draft-p-min 0, 256 tokens greedy: mean acceptance length 6.71 code / 4.11 prose, matching upstream (6.71 / 4.25). Greedy output identical to no-draft on code and short prompts; prose flips one near-tie word at char 470 (batched verification rounding). Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Zihan Zhang <tiancaizhangdaxian@sjtu.edu.cn> Co-authored-by: Jian Chen <jianchen0311@gmail.com> Co-authored-by: Xuan-Son Nguyen <son@huggingface.co> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

SubSir and others added 2 commits August 27, 2026 19:05

revert top-k.cu changes

4e2f54f

github-actions Bot added model Model specific testing Everything test related conversion labels Aug 27, 2026

ngxson marked this pull request as ready for review August 27, 2026 17:16

ngxson requested review from a team, CISC and ggerganov as code owners August 27, 2026 17:16

ngxson merged commit b10f9ca into master Aug 27, 2026
25 of 28 checks passed

sjvrensburg mentioned this pull request Aug 28, 2026

Feature Request: Support for Laguna S 2.1 DFlash #26669

Open

4 tasks done

ggerganov mentioned this pull request Sep 4, 2026

llama.cpp : bump version to 0.4.0 #28386

Merged

YOSSII812001 mentioned this pull request Sep 4, 2026

ローカルAI設定で投機的デコード(speculative decoding)設定を可能にする Robbits-CO-LTD/sprint-coder#434

Closed

tilloh-dev mentioned this pull request Sep 5, 2026

feat(llama.cpp): experimentelles Qwen3.8-27B DFlash2-Preset (200ctx, Q4_K_M Drafter) tilloh-dev/tilloh-backup#9

Merged

5 tasks

FreelineGuide mentioned this pull request Sep 8, 2026

[Feature]: Add community Qwen3.8-27B EfficientThink (anti-overthinking 27B) to the Local Models catalog NousResearch/hermes-agent#106033

Open

1 task

nathansutton mentioned this pull request Sep 11, 2026

benchmarks/stock: llama.cpp with its own DFlash2 drafter (and why it doesn't speed up here) nathansutton/chad#52

Merged

NakliTechie mentioned this pull request Sep 24, 2026

spec : add DFlash2 support (cherry-pick of ggml-org/llama.cpp#27816) PrismML-Eng/llama.cpp#261

Merged

bong-water-water-bong mentioned this pull request Sep 28, 2026

spec: DFlash2 drafters (port of llama.cpp#27816) + HIP top-k for large rows 1bit-MONSTER/ROCmFPX#2

Merged

bong-water-water-bong mentioned this pull request Sep 28, 2026

One server: W4A4 prompts and DFlash2 decode on the lean ROCm route 1bit-MONSTER/engine#194

Merged

github-actions Bot mentioned this pull request Sep 29, 2026

Recipes, tools/bench.py and a KernelForge task: tuned settings as data, every number measured 1bit-MONSTER/engine#233

Merged

Navigation Menu

spec : add DFlash2 support (local convolution + candidate selector) (#27342) - #27816

spec : add DFlash2 support (local convolution + candidate selector) (#27342)#27816
ngxson merged 2 commits into
masterfrom
xsn/dflash2

ngxson commented Aug 27, 2026

Uh oh!

Uh oh!

anfedoro commented Sep 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

ngxson commented Aug 27, 2026

Uh oh!

Uh oh!

anfedoro commented Sep 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants