flake(ci): esp_qemu_runtime probe test fails intermittently on ubuntu — write-then-exec race in a required check

[zackees]

Symptom

Check (ubuntu-latest) failed on PR #1350 with a single test failure in a crate that PR does not touch:

thread 'toolchain::esp_qemu_runtime::tests::probe_linux_exports_the_bundle_on_ld_library_path'
panicked at crates/fbuild-toolchain/src/toolchain/esp_qemu_runtime.rs:525:9:
without the bundle the stub must fail like a real missing .so
test result: FAILED. 141 passed; 1 failed; 1 ignored

Re-running the same job with no code change passed. Evidence it is scheduling-dependent rather than a real regression:

branch SHA	Check (ubuntu-latest)
78d8f27e	success
8dd5aace	success
ef687774	failure
ef687774 (re-run, identical tree)	success

The only delta between 8dd5aace and ef687774 was a HashMap key change in fbuild-build-engine — nothing that can reach an fbuild-toolchain QEMU probe.

Likely mechanism

The test writes a shell script, marks it executable, and immediately executes it:

std::fs::write(&probe, format!("#!/bin/sh\n..."))?;
fbuild_core::platform::fs::set_executable(&probe)?;
// ...
probe_qemu_binary(&probe, None)   // must classify as MissingSharedLibrary

On Linux, execve returns ETXTBSY when any process still holds a writable descriptor for the file. libtest runs cases on parallel threads, and several tests in this binary spawn subprocesses; a concurrent fork in another thread can inherit the still-open write descriptor for this script, so the exec fails for a reason unrelated to the script's contents.

probe_qemu_binary classifies only exit 127 as MissingSharedLibrary. A failed exec produces something else (exit 126, or an Err), so the assertion fails — and it fails with a message about shared libraries, which points the reader at QEMU rather than at process spawning.

Why it matters

This is a default-on unit test in a required status check, so it can redden any PR in the repo at random and send the author looking at code they did not touch. It cost a diagnosis cycle on #1350 for exactly that reason.

Suggested fixes (any one would do)

1. Close the descriptor before exec-ing, and open the file with O_CLOEXEC wherever the harness writes fixtures it intends to run. This is the root fix.
2. Retry once on ETXTBSY inside the probe's spawn path, which is a well-known and legitimate accommodation for this race.
3. Widen the assertion to accept any non-zero exec failure as "not started" — weaker, because the test would then also pass if the script were simply broken.

Worth pairing with (1): probe_qemu_binary currently maps only exit 127 to MissingSharedLibrary and silently lumps every other failure into a different bucket. Distinguishing "the binary could not be executed at all" from "the binary started and could not find a .so" would have made this failure self-describing.

Found while working #1350; not caused by it.