cli & replayer: Prevent Linux worker process orphaning with race-free PR_SET_PDEATHSIG

[louzt]

Background

When a Steam/Proton parent exits abruptly (SIGKILL, crash, game exit), the
forked fossilize_replay / ExternalReplayer worker can end up reparented
to a subreaper (PID 1 in plain hosts, pv-adverb inside pressure-vessel /
srt-bwrap) and keep compiling Vulkan SPIR-V in the background.

Two upstream fixes already exist on this path:

• Terminate replayer processes when parent process (unexpectedly) dies. #96 (2020, HansKristian-Work, merged): close_fd sentinel + setpgid
  • killpg(SIGKILL) in the replayer master. Handles the case where the
    spawned worker shares the parent's process group.
• Add detailed validation of workgroup sizes and limits in CS/MS/TS #233 (Windows-only): Job Objects with JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE.

This PR complements #96 by binding the worker to a kernel-level death
signal (prctl(PR_SET_PDEATHSIG, SIGKILL)) so the cleanup also fires when
the worker ends up in a different process group (the CLI tool path used
when inherit_process_group=true, see cli/fossilize_replay.cpp). It also
adds explicit --device-pci-vendor / --device-pci-device selection so
hybrid laptops can pin shader compilation to the target dGPU instead of
falling through to the iGPU.

Closes: #247, #264, #279, #302.

#288 (104 GB RAM growth) and #303 (status flag) are tracked separately
and are not in scope of this PR — see "Out of scope" below.

---

Relationship with PR #96

flowchart TD
    subgraph PR96["PR #96 (HansKristian-Work, 2020) - User-Space Group Cleanup"]
        P1["Worker spawned"] --> P2{"inherit_process_group?"}
        P2 -- "false (default)" --> P3["close_fd pipe + setpgid()"]
        P3 --> P4["epoll sentinel in master"]
        P4 --> P5{"Parent dies?"}
        P5 -- "Yes, gracefully" --> P6["killpg(SIGKILL) in user-space"]
        P5 -- "SIGKILL / OOM / crash" --> P7["Master code never runs, no killpg, worker orphaned to PID 1"]
        P2 -- "true (CLI path)" --> P8["Pipe and setpgid bypassed, no user-space cleanup at all"]
    end

    subgraph PR305["PR #305 (Louzt, 2026) - Kernel-Level Invariant"]
        Q1["Worker spawned"] --> Q2["prctl(PR_SET_PDEATHSIG, SIGKILL)"]
        Q2 --> Q3["getppid() race-free gate"]
        Q3 --> Q4["Kernel delivers SIGKILL on parent death, any pgroup or namespace"]
    end

    P7 -. "Orphaned to PID 1 or pv-adverb" .-> Q1
    P8 -. "Same orphan path" .-> Q1

Loading

PR #96 works while control returns to the replayer epoll loop. It cannot
cover three cases that PR #305 now does:

1. Involuntary parent death (SIGKILL, Proton panic, OOM killer):
   the master code path is interrupted before killpg is reached, so the
   worker outlives the parent.
2. inherit_process_group=true (CLI tool path, see
   cli/fossilize_replay.cpp): the close_fd pipe is only created when
   !options.inherit_process_group. When the CLI inherits the process
   group, no user-space cleanup runs at all.
3. Container namespaces (pressure-vessel / srt-bwrap): workers
   reparented to pv-adverb (the container subreaper) escape the parent's
   process group, so neither killpg nor setpgid reaches them.

PR #305 binds each worker to a kernel death signal after fork() instead
of relying on user-space event loops, which closes all three gaps.

---

Changes

Linux worker cleanup (fossilize_replay_linux.hpp, fossilize_external_replayer_linux.hpp)

After fork() in both the CLI replayer master and the ExternalReplayer
worker path:

if (prctl(PR_SET_PDEATHSIG, SIGKILL) != 0)
{
    LOGE("Failed to set PR_SET_PDEATHSIG, aborting child startup.\n");
    _exit(EXIT_FAILURE);
}
if (getppid() != parent_pid)
    _exit(EXIT_FAILURE);

The getppid() re-check is the race-free gate: between fork() returning
and prctl taking effect, the parent could already have died and been
reaped. If getppid() != parent_pid, the parent is gone and we abort
cleanly instead of running unparented.

Multi-GPU selection (cli/device.cpp, cli/fossilize_replay.cpp, fossilize_external_replayer.cpp/hpp)

New flags --device-pci-vendor and --device-pci-device (hex, e.g.
0x10de for NVIDIA). Selection is gated on vendorID; the optional
deviceID narrows it to a specific GPU. If the filter is set but no GPU
matches, LOGW warns and falls back to default selection instead of
silently picking the wrong device.

The predicate lives in cli/device.hpp::VulkanDevice::pci_filter_matches
so both the CLI selection loop and the unit test exercise the same logic.

---

Verification

All 8 ctest pass (1.5 s on this host):

1/8 fossilize-system-test ............ Passed
2/8 varint-system-test ............... Passed
3/8 application-info-filter-test ..... Passed
4/8 object-cache-test ................ Passed
5/8 feature-filter-test .............. Passed
6/8 linux-futex-test ................. Passed
7/8 orphan-prevention-test ........... Passed
8/8 gpu-selection-test ............... Passed

• orphan-prevention-test forks an intermediate parent, kills it, and
  asserts the grandchild is reaped within 2 s (bounded poll, 200 × 10 ms).
• gpu-selection-test runs pci_filter_matches against a synthetic
  catalogue of NVIDIA / AMD / Intel IDs and checks vendor-only,
  vendor+device, mismatched device, and unset-filter cases.

The change is Linux-only; Windows behavior is unchanged and still
governed by Job Objects (#233).

---

Out of scope

• Vulkan UMA / VRAM reclamation timing on SIGKILL — relies on the
  kernel's drm_release() path the same way every other Vulkan process
  does. Not benchmarked here.
• macOS — prctl is Linux-only. macOS still uses the PR Terminate replayer processes when parent process (unexpectedly) dies. #96 process-group
  cleanup.
• #288 (104 GB RAM growth): separate pathological pattern — looks like
  compilation state / VM growth rather than orphaning. This PR does not
  address it; a separate report with ps / nvidia-smi output would help
  triage.
• #303 (progress flag): enhancement request, not a bug fix. Not in scope
  of this PR.
• The "% CPU" numbers reported in the original PR description were not
  measured in this PR and have been removed.

[HansKristian-Work]

This overall has a hint of LLM spam that I don't want to engage with, but the prctl idea makes sense.

I merged the prctl part through a squashed down commit on master. For device filtering, that could be a different PR since it's completely unrelated, but the added test coverage is somewhat overzealous and not needed unless I ask for it.

[louzt]

Thanks for pulling the prctl kernel signal approach onto master, glad it helps other users! :)

To provide context on why I structured the PR this way, i really spent time debugging this background process leak on my local hybrid laptop setup (iGPU + dGPU) alongside the community reports/issues open i found useful later on that day, because this sintomps are kinda uncomfortable to deal with as user.

From an outsider perspective trying to deliver this as complete as i can like a production ready fix for end users/or to ppl like you to help me solve it better, so bundling the kernel teardown, PCI device filters, and synthetic tests felt like the most thorough and respectful approach understanding what the Company behind means and that the collaborators had to read.
Quiet a time went into tracing this Linux kernel behavior on my hardware, trying other strats, the usual when you care, not a blind LLM prompt dump spam and token burn and so on, because that would fix this problem while ago, which is why the "don't want to engage" remark caught me off guard, like cmon, im free to PR if i feel something useful on the most useful way my exp can prepare on behalf of my name. But its okey. We're all busy and tired of slop doing all to being less sloppy, and you are the filter.

Im are glad to help providing at least a good minimal triage/poc, so being open-minded about cutting or splitting scope is totally understandable, i was doubting about the proposal reach before even the first commit, so, keeping future contributions strictly atomic/short makes complete sense for upstream maintenance here and i will do that if found other time someday. Here was just matter of me solving my problem until i realize was a good PR op to help others on the road as OpenS and my person asks, i didnt knew u will reject my test because u dont ask for them or like those extra LOC i put around. If useful for others to trim ideas, ok, if not, ok with the trim. The squashed down commit, +10. Use what u really need on your edge.

That said, get it. And sorry for being to -vvvv verbose on the PR body and here haha, just personality preferences (with and without LLM) i guess. Gonna continue outside. Very happy to see my laptop efficient on this edge case. Have a good day mate. Thanks for your time for review and merge.

[HansKristian-Work]

Sorry, didn't mean to sound dismissive. When you're the filter for this stuff long enough, it wears you down and mistakes are made. Thanks for the prctl fix at least.