cargo test renders device unresponsive due to MSVC linker RAM usage #12916

Problem

Invoking cargo t on an i7-13700K Win11 with 32GB RAM makes the whole device unresponsive.

CPU is around 20-30%, RAM climbs at 100% and is likely paging heavily, process explorer shows 1 instance of the MSVC linker (link.exe) per core (24), each consuming 1+ GB of RAM.

Steps

1. Get a project with more doc tests than physical cores, and than GB or RAM. A good example is https://github.com/bevyengine/bevy, my own test is from https://github.com/djeedai/bevy_hanabi.
2. Run cargo t on Windows 11 (not sure version matters; use of MSVC is required though)

Possible Solution(s)

• Limit the number of threads used when cargo test runs rustdoc on Windows, due to the large RAM footprint of the MSVC linker (about 1 GB per test running / link.exe instance). This is likely not ideal because RAM footprint probably depends on crate(s) linked. In the example used (Bevy) the project is very large (300+ packages), so the linker very busy.
• Allow per-project configuring in advance the number of doc tests, for example in Cargo.toml (Enable specifying --test-threads in Cargo.toml #8430)

Notes

Confirmed that cargo t -- --test-threads=8 yields a lot smoother experience, as RAM usage never goes above 60% and therefore CPU usage is much higher (almost CPU bound).

This affects only doc tests, not regular tests (which don't link I guess). Solution ideally shouldn't limit the number of parallel tests for regular tests, only limit parallel linker instances / doc tests.

Somewhat related to #8430 but for different reason.

Version

cargo 1.72.1 (103a7ff2e 2023-08-15)
release: 1.72.1
commit-hash: 103a7ff2ee7678d34f34d778614c5eb2525ae9de
commit-date: 2023-08-15
host: x86_64-pc-windows-msvc
libgit2: 1.6.4 (sys:0.17.2 vendored)
libcurl: 8.1.2-DEV (sys:0.4.63+curl-8.1.2 vendored ssl:Schannel)
os: Windows 10.0.22621 (Windows 11 Professional) [64-bit]