Double mapper is created at a clipped size that cannot be used (RLIMIT_FSIZE), so the existing W^X fallback never runs

[matbrgz]

Description

Follow-up to #117819, fixed by #119316 for 10.0.

#119316 stopped CreateDoubleMemoryMapper() from dying with SIGXFSZ by clipping the ftruncate() size to RLIMIT_FSIZE. It does not, however, tell the caller that the resulting mapper is too small to be used — it clips and returns true, even when the clipped size is 0. The runtime then fails to start with an error that names neither the limit nor the knob that would fix it.

The part that makes this worth reporting is that the fallback already exists and is correct — it is simply never reached. ExecutableAllocator::Initialize():

if (IsDoubleMappingEnabled())
{
    if (!VMToOSInterface::CreateDoubleMemoryMapper(&m_doubleMemoryMapperHandle, &m_maxExecutableCodeSize))
    {
        g_isWXorXEnabled = false;   // <-- run without W^X; this is what DOTNET_EnableWriteXorExecute=0 does
        return true;
    }
    ...

Because CreateDoubleMemoryMapper() reports success, g_isWXorXEnabled stays on, and the failure surfaces much later in ExecutableAllocator::AllocateOffset(), where newFreeOffset > m_maxExecutableCodeSize has no fallback left:

if (newFreeOffset > m_maxExecutableCodeSize)
{
    return false;
}

So a process that is perfectly capable of running (single-mapped) fails to start, and the diagnostic is HRESULT: 0x8007000E.

Reproduction Steps

Any app, any RLIMIT_FSIZE below roughly 3 MiB:

$ ( ulimit -f 0; ./app >/dev/null 2>&1 ); echo $?
137
$ ( ulimit -f 0; ./app )
Failed to create CoreCLR, HRESULT: 0x8007000C

$ ( ulimit -f 0; DOTNET_EnableWriteXorExecute=0 ./app )
Hello, World!

(>/dev/null matters when measuring $? under a small RLIMIT_FSIZE — redirecting to a regular file makes the shell take the SIGXFSZ, which is easy to mistake for the runtime crashing.)

Measured on 10.0.401 (Microsoft.NETCore.App 10.0.12), linux-arm64, on both mcr.microsoft.com/dotnet/sdk:10.0-alpine (musl) and mcr.microsoft.com/dotnet/sdk:10.0 (glibc, Ubuntu 24.04) — identical results:

RLIMIT_FSIZE	default (W^X on)	DOTNET_EnableWriteXorExecute=0
0	exit 137, Failed to create CoreCLR, HRESULT: 0x8007000C	runs
1 MiB	exit 137, Failed to load System.Private.CoreLib.dll (error code 0x8007000E)	runs
2 MiB	exit 134, Out of memory. + Aborted	runs
3 MiB and up	runs	runs

strace at RLIMIT_FSIZE=0 shows the clipped mapper being created and accepted:

memfd_create("doublemapper", MFD_CLOEXEC) = 8
ftruncate(8, 0)                           = 0

against, with no limit:

memfd_create("doublemapper", MFD_CLOEXEC) = 8
ftruncate(8, 8319504384)                  = 0

Expected behavior

When RLIMIT_FSIZE leaves no usable double mapping, CreateDoubleMemoryMapper() returns false and the runtime starts with W^X disabled — the fallback ExecutableAllocator::Initialize() already implements — instead of failing to start.

Actual behavior

The mapper is created at the clipped size (0 bytes when the limit is 0), Initialize() treats that as success, and startup fails later with 0x8007000C / 0x8007000E / Out of memory., naming neither RLIMIT_FSIZE nor DOTNET_EnableWriteXorExecute.

Regression?

Not a regression — it is the residue of the #117819 fix. Before #119316 the same configuration crashed with SIGXFSZ; now it fails to start with an error instead, which is an improvement.

For completeness, measured on the same machine: sdk:9.0-alpine and sdk:8.0-alpine both exit 153 (SIGXFSZ) at ulimit -f 65536 (32 MiB) and at ulimit -f 8192 (4 MiB), and both run when DOTNET_EnableWriteXorExecute=0 is set — i.e. the original crash, as expected, since the fix shipped in 10.0.

Known Workarounds

DOTNET_EnableWriteXorExecute=0, which is what we do in production. It works, but it has to be discovered: nothing in the failure points at it.

Configuration

.NET 10.0.401 / Microsoft.NETCore.App 10.0.12, linux-arm64, Alpine 3.22.6 (musl) and Ubuntu 24.04 (glibc 2.39), under Docker.

Not specific to a configuration as far as I can tell — the clipping code is common to all Unix targets that use the memfd/shm double mapper.

Other information

Why a small RLIMIT_FSIZE is a normal thing to be running under. We run an open-source ICPC-style contest judge: every submission executes under bubblewrap with RLIMIT_FSIZE set, because that is the standard way to stop a contestant's runaway while true: write(...) from filling the judge's disk. RLIMIT_FSIZE=0 is the same idea taken to its end — a process that must not create files at all. #117819 came from a different direction (a site-wide 24 GB limit, and the VS Code C/C++ extension), which suggests the general shape is not rare.

Proposed fix. Return false from CreateDoubleMemoryMapper() when the clipped size cannot back a mapping, so the existing fallback runs. I have opened a PR with the zero case, which is the unambiguous one.

The question I cannot answer from outside: what the minimum should be. Zero is clearly unusable, but the table above shows 1 MiB and 2 MiB are unusable in practice too, and the threshold (~3 MiB here) depends on how much code the runtime maps before it gets to Main. Options seem to be (a) treat only 0 as failure, and leave the small-but-nonzero range failing as it does today, (b) pick a documented minimum below which W^X is disabled, or (c) keep the mapper and let AllocateOffset() failures at startup fall back to single mapping, which is a bigger change. I went with (a) in the PR because it is the one that does not require me to guess a number on your behalf — happy to change it to whatever you prefer.

One latent instance of the original bug, while you are in this file. InitializeTemplateThunkMappingData() does its own memfd_create + ftruncate(fd, MAX_TEMPLATE_THUNK_TYPES * 0x10000) (192 KiB) with no RLIMIT_FSIZE check, and its if (ftruncate(...) == -1) { close(fd); } error path is unreachable under a smaller limit for exactly the reason #117819 described. I could not reach it from a managed app: outside Apple FEATURE_MAP_THUNKS_FROM_IMAGE is not defined, so StubPrecodeCodeTemplate is NULL and CreateTemplate() returns early. So this is not a live crash today as far as I can measure — I mention it only because it is the same pattern in the same file, and it would become one the moment that path is reachable on Linux. Not in the PR; say the word and I will add the guard.