Windows unbuffered model load by JTischbein · Pull Request #26014 · ggml-org/llama.cpp

JTischbein · 2026-07-22T19:56:08Z

Testing

As the DirectIO change on Linux lead to many new issues with several backends and devices, I'd appreciate it if people could test this PR on different devices and let me know if it works.

Overview

Linux supported unbuffered reading from disk via DirectIO for fast model loading. On my machine with PCIe5.0 NVMe drive this path increases the model loading performance from 1-3GB/s to >9GB/s.

For Windows this change is needed, as using mmap of large models on setups with small CPU RAM lead to Sysmem pressure and eviction of runtime pages.

Additional information

This PR introduces the unbuffered read file API of Windows. #18012 is the merged Linux PR.

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: I have used AI for a first implementation. I reviewed, edited and tested the code on my own.

ggml-gh-bot · 2026-07-22T20:00:52Z

Hi @JTischbein, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

Multiple open PRs from a new contributor: We limit new contributors (those without a previously merged PR) to 1 open PR at a time. You currently have 2 open PRs.

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

stevenhoving · 2026-07-23T15:38:22Z

            LocalFree(lpMsgBuf);
        }

+        while (!ret.empty() && (ret.back() == '\r' || ret.back() == '\n')) {


Could you elaborate why this is needed?

The returned message from FormatMessageA often contains a line break at the end. With this change we avoid double line breaks when logging the error message

stevenhoving · 2026-07-23T15:43:32Z


-    impl(const char * fname, const char * mode, [[maybe_unused]] const bool use_direct_io = false) {
-        fp = ggml_fopen(fname, mode);
+    impl(const char * fname, const char * mode, const bool use_direct_io = false) : fname(fname) {


My 2 cents on this mr, is that it actually needs to be decomposed into separate classes. With an interface and multiple implementation classes. This allow you to keep things simple and seperate (mmap, direct_io, win_unbuffered, etc).

I agree with you on that, for now I wanted to keep the changes minimal to validate whether unbuffered IO is welcome on WIndows and works on all systems.

Let me try to get a new version together

I would prefer to make a separate PR for the abstraction of the paths. The required changes are quite large

ORippler

Thanks for bringing this feature to Windows as well!

Regarding the refactor: nice to have, but beyond the scope of this PR imo - this PR aims to bring Windows to performance parity with Linux for the direct-io path
What would be nice is some tests for file loading (both for Linux and Windows). Not sure how difficult they are to spin up and if they should be part of this PR.
Do you have some perf numbers for dGPU systems you could share?

ORippler · 2026-07-27T09:41:07Z

+        void * raw_buffer = _aligned_malloc(bytes_to_read, alignment);
+        if (raw_buffer == nullptr) {
+            LLAMA_LOG_WARN("%s: Falling back to buffered I/O due to %s\n",
+                           __func__, GetErrorMessageWin32(ERROR_NOT_ENOUGH_MEMORY).c_str());
+            CloseHandle(fp_win32_direct);
+            fp_win32_direct = INVALID_HANDLE_VALUE;
+            alignment = 1;
+            seek(offset, SEEK_SET);
+            read_raw_unsafe(dest, len);
+            return;
+        }
+
+        struct aligned_buffer_deleter {
+            void operator()(void * p) const { _aligned_free(p); }
+        };
+        std::unique_ptr<void, aligned_buffer_deleter> buffer(raw_buffer);


There is no need to materialize raw_buffer, just pass it into the unique_ptr directly

Changed with the upcoming commit, thanks!

ORippler · 2026-07-27T09:49:34Z

+        }
+        const size_t bytes_to_read = (offset_from_alignment + len + alignment - 1) & ~(alignment - 1);
+
+        void * raw_buffer = _aligned_malloc(bytes_to_read, alignment);


I guess adding a chunked read is not advisable from a perf perspective? LM Head may be a couple of 100 MB in size (500 MB for Qwen3.6-35BA3B if we have 8 BPW)

Actually we sometimes have regressing performance with larger read size, in my tests above 512MB (1GB compared to 512MB loads 16% slower). I will add chunked reads in the upcoming PR

I can remember a 'trick' to deal with this more efficiently.

first check if the buffer given is already aligned.

If not, we read unaligned until we are at the aligned boundary.

Handle the left over bytes as an aligned read.

That would avoid the memory allocation

nibor1896 · 2026-08-04T08:23:29Z

Just FYI, no critic at all or anything like that:

PR#26542 covers the "OVERLAPPED" and "Handle/Thread" part of this problem - both PRs combined, should solve this problem.

Shared handle at depth 8: 1.01x
One handle per thread: 2.22x

Yours
Robin

ggerganov · 2026-08-06T08:06:31Z

What would be nice is some tests for file loading (both for Linux and Windows). Not sure how difficult they are to spin up and if they should be part of this PR.

Yes, it shouldn't be difficult to unit-test the llama_file and llama_mmap classes:

Generate some large dummy file with known bytes
Read it in different models
Validate read contents are expected
Track performance

Then we can run these tests on multiple devices on master and compare the results after the changes. This way there is no second-guessing.

ORippler · 2026-10-01T08:52:09Z

We should add buffered fall-back path similar to #29749 for Windows also

JTischbein marked this pull request as ready for review July 23, 2026 13:46

JTischbein requested a review from ggerganov as a code owner July 23, 2026 13:46

stevenhoving reviewed Jul 23, 2026

View reviewed changes

taronaeo added help wanted Needs help from the community windows Issues specific to Windows labels Jul 27, 2026

ORippler reviewed Jul 27, 2026

View reviewed changes

This was referenced Aug 3, 2026

Unbuffered positional read on Windows #26541

Closed

llama : add unbuffered positional read on Windows #26542

Open

JTischbein added 3 commits September 3, 2026 14:19

Windows unbuffered io

53d6ccc

Fix read len calculation

93ffd66

Add test and model load timing

a2923f6

JTischbein force-pushed the windows_unbuffered_model_load branch from 2cd2f5f to a2923f6 Compare September 3, 2026 13:48

JTischbein requested a review from CISC as a code owner September 3, 2026 13:48

github-actions Bot added the testing Everything test related label Sep 3, 2026

Conversation

JTischbein commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Testing

Overview

Additional information

Requirements

Uh oh!

ggml-gh-bot Bot commented Jul 22, 2026

Uh oh!

stevenhoving Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

JTischbein Aug 1, 2026

Choose a reason for hiding this comment

Uh oh!

stevenhoving Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JTischbein Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

JTischbein Aug 1, 2026

Choose a reason for hiding this comment

Uh oh!

ORippler left a comment

Choose a reason for hiding this comment

Uh oh!

ORippler Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

JTischbein Aug 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ORippler Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

JTischbein Aug 1, 2026

Choose a reason for hiding this comment

Uh oh!

stevenhoving Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nibor1896 commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ggerganov commented Aug 6, 2026

Uh oh!

ORippler commented Oct 1, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

JTischbein commented Jul 22, 2026 •

edited

Loading

stevenhoving Jul 23, 2026 •

edited

Loading

JTischbein Aug 1, 2026 •

edited

Loading

stevenhoving Aug 7, 2026 •

edited

Loading

nibor1896 commented Aug 4, 2026 •

edited

Loading