openvino: match the MoE fusion on the rank-3 stateful graph

[cavusmustafa]

FuseMoeCompressedFusedGateUp only matched the rank-4 graph, so under GGML_OPENVINO_STATEFUL_EXECUTION=1 gemma-4 26B-A4B silently ran the unfused MoE path: pp512 dropped from ~1600 t/s to 66 t/s while decode kept its gain.

The pattern did not bind at all (the callback was never invoked), because the rank-3 graph differs in node structure, not only in rank:

rank-4: GatherMatmul -> Transpose -> Unsqueeze -> [Convert] -> Slice
rank-3: GatherMatmul -> Convert -> Transpose -> Slice

The rank-3 graph never drops to a batch dim, so it has no Unsqueeze after the expert GEMM - that Unsqueeze sits between the GEGLU and the down projection instead - and its Convert is on the other side of the Transpose. Wrap both in optional<>, which matches either form, in the gate/up and down branches.

With the pattern binding, the routing guard is reached and rejects rank 3. Routing arrives as [n_tokens, top_k, 1] there versus [1, n_tokens, top_k, 1] on the rank-4 graph, so normalise it up to the rank-4 form, keep the op config that the stateless path already validates, and squeeze the batch dim back off the result. Guard that the replacement shape matches the ReduceSum it replaces.

26B-A4B on Arc B390, GPU, GGML_OPENVINO_REQUANT_KQUANT=q4_asym64_all: stateful pp512 66 -> 1585 t/s, tg128 30.2; prefill now at parity with stateless (1609) instead of 24x behind it.

The stateless path is unchanged: 26B perplexity is bit-identical before and after (7990.0534 +/- 1274.45692), and test-backend-ops -o MUL_MAT_ID on OPENVINO0 goes from 575/577 to 577/577.

Overview

Additional information

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: