ggml: add moe_sum operator for efficient MoE expert aggregation

[blime4]

Add a new GGML_OP_MOE_SUM operator that efficiently aggregates outputs
from multiple experts in Mixture of Experts (MoE) models.

Input: [hidden_dim, n_expert_used, n_tokens]
Output: [hidden_dim, n_tokens]

Performance

Benchmark on Qwen3-30B-A3B (Q4_K_M) with NVIDIA A40 (At the moment I only have an A40 graphics card):

Test	ggml_moe_sum	ggml_add loop	Improvement
pp256	145.42 t/s	143.90 t/s	+1.06%
tg512	137.81 t/s	135.85 t/s	+1.44%

Implementation

• CPU: Optimized cache-friendly loop order
• CUDA: Specialized kernels for different workloads
• Automatic fallback to ggml_add loop for non-contiguous tensors
• Runtime disable: LLAMA_DISABLE_MOE_SUM=1

[CISC]

Repeating adds are already fused on most backends, including CUDA:

llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu

Lines 3624 to 3653 in 3795cc1

	if (node->op == GGML_OP_ADD) {
	int n_fuse = 0;
	ggml_op ops[8];
	std::fill(ops, ops + 8, GGML_OP_ADD);
	for (; n_fuse <= 6; ++n_fuse){
	if (!ggml_can_fuse(cgraph, i + n_fuse, ops + n_fuse, 2)) {
	break;
	}
	if (cgraph->nodes[i + n_fuse] != cgraph->nodes[i + n_fuse + 1]->src[0]) {
	break;
	}
	if (!ggml_are_same_layout(cgraph->nodes[i + n_fuse]->src[1], cgraph->nodes[i + n_fuse + 1]->src[1])) {
	break;
	}
	}
	n_fuse++;
	if (n_fuse > 1) {
	for (int j = 0; j < n_fuse - 1; ++j) {
	node->src[j + 2] = cgraph->nodes[i + j + 1]->src[1];
	}
	cgraph->nodes[i + n_fuse - 1]->data = node->data;
	ggml_cuda_op_fused_add(*cuda_ctx, node, n_fuse);
	i += n_fuse - 1;
	continue;
	}
	}

[am17an]

Also this was attempted in before #16857 and it leads to various problems I haven't diagnosed yet. In any case, this should be a fusion optimization not a full fledged ggml operator.

[blime4]

Thank you both @CISC and @am17an for taking the time to review and for your helpful feedback!

Also, regarding @am17an your comment that "this should be a fusion optimization not a full fledged ggml operator" - I'm still learning the codebase architecture and would really appreciate your guidance on this. What exactly makes a "fusion optimization" different from a "full fledged ggml operator" in the llama.cpp design philosophy? And why would the fusion approach be preferred here? I want to make sure I understand the proper way to contribute to this project.

[ngxson]

IMO calling it "moe_sum" is misleading because it has nothing to do with MoE. Indeed, this is just the equivalent of pytorch sum() with dim=1: [x, y, z] -> [x, 1, z]

Probably it's more useful to have an operator sum_dim, or better, extend the current sum_rows to accept custom dim