Qwen3.5 9B MTP cold load fails on RTX 3090 Ti with 4096-token CUDA micro-batch #1200

Bug

Reported by: @joelagnel

The Qwen3.5 9B MTP recipe introduced in #1178 fails its first cold model load on an NVIDIA GeForce RTX 3090 Ti when both --batch-size and --ubatch-size are set to 4096.

The primary model context is created successfully, but creation of the MTP draft context fails while attempting to reserve a second approximately 2.2 GiB CUDA_Host compute buffer. The setup inference smoke then receives HTTP 500 and rolls back the otherwise clean Local AI installation.

This was reproduced from PR #1178 at commit
31dffdb0dbd9eebc8b98ce35cf13ab60762410a9.

Reproduction

1. Use the Local AI implementation from feat: add managed local AI with llama-server #1178 at the commit above.
2. Run a clean Windows Companion onboarding on the RTX 3090 Ti described below.
3. Select qwen3.5-9b-mtp-q4-k-m.
4. Allow the Companion to install llama-server b10488 CUDA 13 x64 and download the pinned Qwen3.5 9B Q4_K_M model.
5. Run the first inference verification with the recipe-generated arguments:

--alias qwen3.5-9b-mtp-q4-k-m
--ctx-size 262144
--batch-size 4096
--ubatch-size 4096
--parallel 1
--n-gpu-layers all
--spec-type draft-mtp
--spec-draft-n-max 3

Expected behavior

The Qwen3.5 9B model and its MTP draft context should load on a supported 24 GiB discrete GPU, and the first inference verification should complete without reducing the model's full 262,144-token context.

Actual behavior

llama-server fully offloads the model and creates the primary context:

CUDA0   : NVIDIA GeForce RTX 3090 Ti
          24563 MiB total, 23285 MiB free

CPU     : 32747 MiB total, 26981 MiB free

offloaded 34/34 layers to GPU

CUDA0 model buffer size       = 5040.86 MiB
CUDA_Host model buffer size   =  545.62 MiB
CUDA0 primary KV buffer size  = 8192.00 MiB
CUDA0 recurrent buffer size   =  201.00 MiB
CUDA0 compute buffer size     = 2752.13 MiB
CUDA_Host compute buffer size = 2176.14 MiB

It then creates the 1,024 MiB MTP KV cache but fails while reserving the MTP compute workspace:

CUDA0 MTP KV buffer size = 1024.00 MiB

ggml_backend_cpu_buffer_type_alloc_buffer:
failed to allocate buffer of size 2281848864

ggml_gallocr_reserve_n_impl:
failed to allocate CUDA_Host buffer of size 2281848864

graph_reserve: failed to allocate compute buffers
llama_init_from_model: failed to initialize the context:
failed to allocate compute pp buffers
common_speculative_init_result: failed to create MTP context
srv load_model: failed to create MTP context

The Companion's setup smoke receives HTTP 500 and rolls back.

Environment

• GPU: NVIDIA GeForce RTX 3090 Ti
• Reported GPU memory: 24,563 MiB
• Free GPU memory before model load: 23,285 MiB
• NVIDIA driver: 591.86
• CUDA runtime: b10488-cuda13-x64
• Model: Qwen3.5 9B Q4_K_M
• Model size: 5,868,826,976 bytes
• Model context: 262,144 tokens
• System memory: 32,747 MiB
• Free system memory before model load: 26,981 MiB
• OS: Windows 11, version 10.0.26200.0
• CPU: Intel Xeon Gold 6154, 8 guest vCPUs
• Virtualization: KVM guest with the complete RTX 3090 Ti assigned through VFIO PCI passthrough
• OpenClaw Windows Node: feat: add managed local AI with llama-server #1178 at 31dffdb0

Why this appears to fail

This was not caused by another process consuming most of the GPU memory: approximately 95% of VRAM was free before the cold load.

The direct failing allocation is CUDA_Host, not CUDA0. With a 4096-token physical micro-batch, the primary context reserves approximately 2.13 GiB of page-locked host compute memory. The MTP draft context then attempts to reserve another allocation of almost exactly the same size, and that second allocation fails.

This makes the recipe's peak transient memory requirement substantially larger than the model weights and KV caches alone suggest. The precise reason Windows/CUDA rejects the second pinned allocation may involve page-locked-memory limits, commit pressure, fragmentation, or driver allocator behavior; the available log does not distinguish among those causes.

A separate #1178 test report indicates that the 4096 value has succeeded in another environment. Therefore, this should be treated as a hardware/environment-sensitive recipe qualification problem rather than a universal failure of Qwen3.5 or MTP.

Potential fix

A locally tested workaround makes the Qwen3.5 9B recipe use a 512-token physical micro-batch while retaining:

• The full 262,144-token context.
• The 4096-token logical batch.
• Full CUDA layer offload.
• MTP speculative decoding.
• A single parallel request.

Example implementation:

diff --git a/src/OpenClaw.Shared/Inference/Catalog/LocalModelCatalog.cs b/src/OpenClaw.Shared/Inference/Catalog/LocalModelCatalog.cs
@@
                 Recipe(
                     fullAttentionLayerCount: 8,
                     keyValueHeadCount: 4,
-                    temperature: 1.0),
+                    temperature: 1.0,
+                    microBatchTokens: 512),
@@
     private static LocalModelRunRecipe Recipe(
         int fullAttentionLayerCount,
         int keyValueHeadCount,
-        double temperature) =>
+        double temperature,
+        int microBatchTokens = 4_096) =>
         new(
             contextTokens: NativeContextTokens,
             keyCachePrecision: KvCachePrecision.F16,
             valueCachePrecision: KvCachePrecision.F16,
             batchTokens: 4_096,
-            microBatchTokens: 4_096,
+            microBatchTokens: microBatchTokens,

A minimal recipe regression could accompany it:

[Fact]
public void Qwen9BRecipe_BoundsCudaMicroBatchForDiscreteGpuColdStart()
{
    LocalModelInfo model =
        LocalModelCatalog.Find(LocalModelCatalog.Qwen9BModelId)!;

    Assert.Equal(512, model.Recipe.MicroBatchTokens);
    Assert.Equal(4_096, model.Recipe.BatchTokens);
    Assert.Equal(
        LocalModelCatalog.NativeContextTokens,
        model.Recipe.ContextTokens);
    Assert.Equal(
        SpeculativeDecodingMode.DraftMtp,
        model.Recipe.SpeculativeDecoding);
}

This regression verifies the recipe but is not sufficient by itself. Qualification should include a real cold model load and first inference with MTP enabled on a supported 24 GiB discrete GPU.

An adaptive micro-batch based on qualified SKU or available allocation headroom may ultimately be preferable. The value 512 is presented as a demonstrated safe value for this RTX 3090 Ti environment, not necessarily the optimal value for every GPU.

Verification of the potential fix

With --ubatch-size 512:

• The same model loaded successfully.
• The full 262,144-token context was retained.
• MTP remained enabled.
• Clean onboarding completed.
• The WSL Gateway used the Windows-native server as its primary inference endpoint.
• A direct primary chat returned the expected response.
• nvidia-smi identified llama-server.exe using 16,006 MiB during inference.
• Observed inference telemetry reached 85% GPU utilization and 365 W board power.

Suggested acceptance criteria

• Cold-load Qwen3.5 9B with MTP on a qualified 24 GiB discrete GPU.
• Preserve the full 262,144-token model context.
• Preserve the 4096 logical batch unless separate testing justifies changing it.
• Bound or adapt the physical micro-batch to avoid the duplicated CUDA_Host allocation failure.
• Complete the first inference smoke without HTTP 500 or setup rollback.
• Record free VRAM and relevant host-memory state during qualification.
• Add a recipe regression plus a real runtime cold-load test.