Skip to content

models: remove redundant ggml_cont calls now supported by CUDA unary kernels - #30198

Merged
CISC merged 4 commits into
ggml-org:masterfrom
saady789:cuda-remove-qwen-cont
Oct 13, 2026
Merged

CISC merged 4 commits into
ggml-org:masterfrom
saady789:cuda-remove-qwen-cont

Conversation

@saady789

@saady789 saady789 commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

Overview

This is a follow-up to #29781. That PR added support for unary operations on non-contiguous tensors in CUDA. With that support now upstream, this PR cleans up redundant ggml_cont / ggml_cont_2d allocations and their associated TODO comments across affected models.

Removing these intermediate copies simplifies the compute graphs and avoids unnecessary memory bandwidth overhead during the forward pass.

Changes

  • src/models/delta-net-base.cpp:
    • Removed redundant ggml_cont on g_last before ggml_exp in build_delta_net_chunking.
  • src/models/qwen3next.cpp:
    • Removed ggml_cont_2d on gate before ggml_sigmoid in build_layer_attn.
    • Removed ggml_cont on b before ggml_sigmoid in build_layer_attn_linear.
    • Removed ggml_cont_2d on gate before ggml_sigmoid in graph_mtp::graph_mtp.

Requirements

@saady789
saady789 requested a review from CISC as a code owner October 9, 2026 03:15
@github-actions github-actions Bot added the model Model specific label Oct 9, 2026
@CISC

CISC commented Oct 9, 2026

Copy link
Copy Markdown
Member

@CISC

CISC commented Oct 9, 2026

Copy link
Copy Markdown
Member

Also, there are more that are not marked with TODO, f.ex. in dflash:

cat_conf = ggml_sigmoid(ctx0, ggml_cont(ctx0, ggml_view_2d(ctx0, base, 1, n_blocks, base_stride, 0)));

@saady789

Copy link
Copy Markdown
Contributor Author

Also, there are more that are not marked with TODO, f.ex. in dflash:

cat_conf = ggml_sigmoid(ctx0, ggml_cont(ctx0, ggml_view_2d(ctx0, base, 1, n_blocks, base_stride, 0)));

Removed the redundant ggml_cont call in src/models/dflash.cpp.

Regarding src/models/qwen3next.cpp: kept ggml_cont_2d in place because the subsequent ggml_mul invokes ggml_mul_impl, which asserts GGML_ASSERT(ggml_can_repeat(b, a)). Without standard contiguous layout here, the repeat/broadcasting check fails. Verified the graph execution passes on Tesla T4.

Comment thread src/models/qwen3next.cpp Outdated
Comment thread src/models/qwen3next.cpp Outdated
saady789 and others added 2 commits October 12, 2026 23:51
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
@CISC
CISC merged commit 071b553 into ggml-org:master Oct 13, 2026
17 of 18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

model Model specific

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants