Skip to content

cuda: refuse unsafe host fallback during SSD streaming - #737

Open
exochard wants to merge 1 commit into
antirez:mainfrom
exochard:fix/cuda-host-pointer-fallback
Open

cuda: refuse unsafe host fallback during SSD streaming#737
exochard wants to merge 1 commit into
antirez:mainfrom
exochard:fix/cuda-host-pointer-fallback

Conversation

@exochard

@exochard exochard commented Aug 7, 2026

Copy link
Copy Markdown

Problem

DeepSeek V4 Flash 0731 is an approximately 80 GiB GGUF, so the CUDA path must use SSD streaming on a 16 GiB RTX 4060 Ti. When the bounded device range cache or arena could not satisfy a request, the generic resolver could fall through to cuda_model_ptr() and return a host mmap address to a CUDA kernel. Under sustained layer traffic that becomes host/device pointer confusion and can surface as an illegal memory access.

Change

  • Give the generic CUDA weight cache a conservative default ceiling based on live free VRAM when no explicit limit is configured.
  • Make SSD streaming authoritative: disable raw HMM/direct host-pointer paths while it is active.
  • Return a hard refusal after the bounded cache and arena paths reject a streamed range instead of handing the kernel a host pointer.
  • Keep whole-model prefetch/copy disabled in SSD mode so it cannot bypass the bounded streaming path.

The refusal is deliberate: a failed streamed weight lookup is diagnosable and recoverable by the caller; a host address reaching a CUDA kernel is silent memory corruption or an illegal access.

Validation

Builds completed with no compiler warnings:

make -B ds4 CUDA_HOME=/opt/cuda CUDA_ARCH=sm_86
make -B ds4 CUDA_HOME=/opt/cuda CUDA_ARCH=sm_89

git diff --check is clean. The build target was an RTX 4060 Ti 16 GiB (sm_89), with the DeepSeek V4 Flash 0731 IQ2XXS SSD-streamed GGUF. End-to-end CUDA execution was not available in this coding environment; the runtime reproduction and follow-up smoke tests were performed from the user's driver-visible shell on the integration branch.

Scope

This is the safety boundary only. Async uploads and persistent expert reuse are separate follow-up changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant