Fail double mapper creation when RLIMIT_FSIZE leaves no usable size

[matbrgz]

Fixes #134524.

CreateDoubleMemoryMapper() clips the double mapping to RLIMIT_FSIZE (#119316, which fixed #117819), but it reports success even when the clipped size is 0 — which is what a process running under RLIMIT_FSIZE=0 gets. That matters because of who the caller is:

if (IsDoubleMappingEnabled())
{
    if (!VMToOSInterface::CreateDoubleMemoryMapper(&m_doubleMemoryMapperHandle, &m_maxExecutableCodeSize))
    {
        g_isWXorXEnabled = false;   // run single mapped
        return true;
    }

ExecutableAllocator::Initialize() already knows how to continue without W^X, and only does it when this function fails. Reporting success with a zero sized mapper skips that, and the failure resurfaces later in AllocateOffset() (newFreeOffset > m_maxExecutableCodeSize), where there is no fallback left — so a process that can run perfectly well single mapped fails to start instead, with HRESULT: 0x8007000C / 0x8007000E.

This change fails mapper creation when the clipping leaves nothing to map, so the existing fallback runs.

Measured (shipped 10.0.401 / Microsoft.NETCore.App 10.0.12, linux-arm64, Alpine 3.22 musl and Ubuntu 24.04 glibc — identical on both)

RLIMIT_FSIZE	default	DOTNET_EnableWriteXorExecute=0
0	exit 137, Failed to create CoreCLR, HRESULT: 0x8007000C	runs
1 MiB	exit 137, Failed to load System.Private.CoreLib.dll (0x8007000E)	runs
2 MiB	exit 134, Out of memory.	runs
3 MiB+	runs	runs

strace at RLIMIT_FSIZE=0 shows the zero sized mapper being created and accepted: memfd_create("doublemapper", MFD_CLOEXEC) = 8 then ftruncate(8, 0) = 0. With this change that path returns false instead, which is the same state the last column reaches by hand.

Scope, and what I deliberately left out

This only covers the unambiguous case — a mapper that cannot back a single allocation. The table shows 1 MiB and 2 MiB are unusable in practice too, but the threshold depends on how much code the runtime maps before Main, and picking a minimum is a judgement call that belongs to you rather than to me; #134524 lays out the options. Happy to extend this PR in whichever direction you prefer.

No behaviour change when RLIMIT_FSIZE is unlimited or large enough, which is every normal configuration.

I am building clr+libs locally to confirm the end-to-end behaviour on linux-arm64 and will post the result as a comment — the reasoning above is from reading the code and measuring the shipped runtime, not yet from a build with this patch in it.

[azure-pipelines]

Azure Pipelines:
Successfully started running 3 pipeline(s).
13 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[dotnet-policy-service]

Tagging subscribers to this area: @agocke
See info in area-owners.md if you want to be subscribed.

[matbrgz]

Verified end to end on a local build, as promised in the description.

Build: main @ 696cea02, ./build.sh clr+libs+host -c Release, linux-musl-arm64 (Alpine 3.22), then the same tree rebuilt with only this patch applied. Same console app for every row (Hello, World!, built with the 10.0.401 SDK, run through the testhost with DOTNET_ROLL_FORWARD=LatestMajor).

RLIMIT_FSIZE	W^X	before this PR	with this PR
0	on (default)	exit 137, Failed to create CoreCLR, HRESULT: 0x8007000C	exit 0, Hello, World!
0	off	exit 0	exit 0
1 MiB	on	exit 137, Failed to load System.Private.CoreLib.dll (0x8007000E)	exit 137, unchanged
1 MiB	off	exit 0	exit 0
unlimited	on / off	exit 0	exit 0

strace of the patched runtime at RLIMIT_FSIZE=0 shows the path the change intends — the mapper is opened, the new check returns before any ftruncate, and ExecutableAllocator::Initialize() continues single-mapped:

memfd_create("doublemapper", MFD_CLOEXEC) = 6
(no ftruncate)
Hello, World!
rc=0

Before the patch the same trace showed ftruncate(fd, 0) = 0 followed by the startup failure.

Two notes on scope, so the table is not read as more than it is:

• The 1 MiB row is unchanged on purpose. This PR only handles the size that cannot back any allocation; the small-but-nonzero range is the open question in Double mapper is created at a clipped size that cannot be used (RLIMIT_FSIZE), so the existing W^X fallback never runs #134524, and the table confirms it still needs a decision.
• A reproduction pitfall I hit myself: after rebuilding only clr, the testhost under artifacts/bin/testhost/.../shared/Microsoft.NETCore.App/<ver>/ still carried the previous libcoreclr.so (different hash, older timestamp). My first "patched" run therefore measured the unpatched binary and looked like a no-op. The table above is from a run after copying the rebuilt libcoreclr.so into the testhost (sha256 prefix 027b1bdb…, vs. 4dd2e7f3… for the stock one). Mentioning it in case anyone re-runs this and sees no difference.