Skip to content

compile_example hangs when called concurrently from parallel tests #434

@basil

Description

@basil

When multiple test threads call compile_example concurrently, each invocation spawns a separate cargo build --example subprocess. These subprocesses all attempt to acquire an exclusive flock on the build directory (Filesystem::open_rw_exclusive_create), and the parent cargo test process holds a lock on the same directory that won't be released until the tests complete.

Process tree when hung

      214795   /usr/bin/python3 -sP /usr/bin/fedpkg --release f43 local
        214821   rpmbuild --define _sourcedir src/fedora/rust-ptools --define _specdir ...
          217222   /bin/sh -e /var/tmp/rpm-tmp.oVUWqh
            217223   /usr/bin/cargo test -j32 -Z avoid-dev-deps --profile rpm --no-fail-fast
              218505    src/fedora/rust-ptools/rust-ptools-0.2.21-build/ptools-0.2.21/target/rpm/deps/pargs_penv_test-108be
                218512   /usr/bin/cargo build --message-format=json --target x86_64-unknown-linux-gnu --release --example pargs_penv
                218514   /usr/bin/cargo build --message-format=json --target x86_64-unknown-linux-gnu --release --example pargs_penv
                218515   /usr/bin/cargo build --message-format=json --target x86_64-unknown-linux-gnu --release --example pargs_penv
                218517   /usr/bin/cargo build --message-format=json --target x86_64-unknown-linux-gnu --release --example pargs_penv
                218518   /usr/bin/cargo build --message-format=json --target x86_64-unknown-linux-gnu --release --example penv_setenv
                218520   /usr/bin/cargo build --message-format=json --target x86_64-unknown-linux-gnu --release --example pargs_penv
                218521   /usr/bin/cargo build --message-format=json --target x86_64-unknown-linux-gnu --release --example pargs_penv
                218522   /usr/bin/cargo build --message-format=json --target x86_64-unknown-linux-gnu --release --example pargs_penv
                218523   /usr/bin/cargo build --message-format=json --target x86_64-unknown-linux-gnu --release --example penv_setenv

Stack trace of blocked subprocess

0x00007f2b53d7a67b flock+0xb
0x00005619ac359db1 <cargo::util::flock::Filesystem>::open_rw_exclusive_create::<alloc::string::String>::{closure#1}+0x11
0x00005619ac690069 cargo::util::flock::acquire+0x269
0x00005619abf95237 <cargo::util::flock::Filesystem>::open_rw_exclusive_create::<&str>+0xd7
0x00005619ac419d27 <cargo::core::compiler::layout::Layout>::new+0x9f7
0x00005619ac40bb2d <cargo::core::compiler::build_runner::BuildRunner>::prepare_units+0x2bd
0x00005619ac413216 <cargo::core::compiler::build_runner::BuildRunner>::compile+0x4d6
0x00005619ac577ca0 cargo::ops::cargo_compile::compile_ws+0xc90
0x00005619ac57c391 cargo::ops::cargo_compile::compile_with_exec+0x51
0x00005619ac583abb cargo::ops::cargo_compile::compile+0x5b
0x00005619ac90f929 cargo::commands::build::exec+0x4b9
0x00005619ac8e69df <cargo::cli::Exec>::exec+0x55f
0x00005619ac8e304f cargo::main+0x37ef
0x00005619ac8be593 std::sys::backtrace::__rust_begin_short_backtrace::<fn(), ()>+0x3
0x00005619ac8d28a9 std::rt::lang_start::<()>::{closure#0}+0x9
0x00005619ace65ca6 std::rt::lang_start_internal::h31d4445a63e30e2c+0x4b6
0x00005619ac95271c main+0x2c
0x00007f2b53c995b5 __libc_start_call_main+0x75
0x00007f2b53c99668 __libc_start_main@@GLIBC_2.34+0x88
0x00005619abef34c5 _start+0x25

Workaround

Use compile_examples (plural) in a LazyLock to build all examples with a single cargo build invocation, avoiding concurrent lock acquisition:

static EXAMPLES: LazyLock<HashMap<String, PathBuf>> = LazyLock::new(|| {
    snapbox::cmd::compile_examples(["-q"])
        .expect("failed to compile examples")
        .map(|(name, path)| (name, path.expect("failed to compile example")))
        .collect()
});

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions