doublemapping.cpp causes crash with SIGXFSZ if file size limits <2TB are in effect

[djast]

Description

src/coreclr/minipal/Unix/doublemapping.cpp does:

if (ftruncate(fd, MaxDoubleMappedSize) == -1)

where

#ifdef TARGET_64BIT
static const off_t MaxDoubleMappedSize = 2048ULL10241024*1024;
#else
static const off_t MaxDoubleMappedSize = UINT_MAX;
#endif

This causes the process to fail with SIGXFSZ if a file size limit has been imposed via setrlimit()/prlimit().

Reproduction Steps

Set a file size limit (e.g., "ulimit -S -f 25165824" from bash).
Run any program which invokes VMToOSInterface::CreateDoubleMemoryMapper().

Expected behavior

Process continues normally.

Actual behavior

Process dies with SIGXFSZ.

Regression?

Issue was observed in the Visual Studio Code C/C++ extension; problem occurs in ms-vscode.cpptools-1.23.5-linux-x64 but was not present in ms-vscode.cpptools-1.22.11-linux-x64 (cf. microsoft/vscode-cpptools#13768).
See also microsoft/vscode#251037 , in VS Code's vsce-sign program.
I presume these projects are embedding their libmscordaccore.so libraries from dotnet/runtime.

Known Workarounds

No response

Configuration

AlmaLinux 8.10 x86_64.

Other information

Issue could be addressed by calling getrlimit(RLIMIT_FSIZE) and capping the parameter to ftruncate() at the soft limit.

[janvorli]

@djast what is the limit that your device has set? I'd like to understand how low it actually is. If it is smaller than the size of all the executable code loaded from crossgened assemblies, generated by JIT and some runtime helpers, then it would prevent your app execution even if we dynamically set the double mapping max size based on the limit.
Having said that, I believe we should honor that limit.

[djast]

At our site, we currently set RLIMIT_FSIZE to 24GB to prevent users from inadvertently filling up disk partitions with runaway processes (or creating huge sparse files that may interfere with backup/restore processes).

If the size of all the executable code exceeds that limit, the program isn't running anyway, because the VM subsystem will be paging itself into oblivion.

I don't know whether the 2TB anonymous memory allocation may have other adverse performance effects on some systems (e.g., systems which disable or limit memory overcommitment), but it may be prudent to consider reducing the allocation to something more reasonable (e.g., the amount of virtual memory in the system) even if RLIMIT_FSIZE does not prevent creating memfd's of unlimited size.

[janvorli]

but it may be prudent to consider reducing the allocation to something more reasonable (e.g., the amount of virtual memory in the system)

The information I have found about what influences the max memfd size seems to indicate that while the total mapped size is limited, I have found no information on limits of the total memfd size besides the RLIMIT_FILESIZE. I think we might set the limit to say twice the amount of the physical memory to accommodate for the theoretical extreme case when all the physical memory would be double mapped as RW and RX at the same time. And of course cap that by the RLIMIT_FILESIZE.