fix(shim): never let a tracer-attach failure stop the real binary from running - #159
Conversation
link.Tracepoint("sched", "sched_process_fork") is the only attach in
Start() that goes through perf_event_open(2) instead of bpf(2). A
seccomp-sandboxed caller commonly permits the latter and withholds the
former -- systemd-udevd's SystemCallFilter lists "bpf" but not
"perf_event_open" -- so this one call fails where every uprobe attaches
fine.
Treating that as fatal cost far more than the tracepoint is worth: the
helper returned an error, runWithTracing propagated it, and main() exited
before ever exec'ing the real binary. A shimmed udev callout was therefore
replaced by a stub that always failed. On openSUSE that silently disabled
NFS readahead tuning for as long as the shim stayed installed, on top of
reporting 0% coverage for a binary that never got the chance to run.
The tracepoint only widens the watched set to child processes, so losing
it costs coverage of forks, never of the process we were asked to trace.
Degrade to a debug log instead.
Refs #158
runWithTracing returned an error whenever the helper failed to report ready, and main() turns that into os.Exit(1). Because that happens before syscall.Exec(realBin), an environment that blocks eBPF setup did not just lose coverage -- it replaced the instrumented program with a stub that always failed, for as long as the shim stayed installed. Observed on udev's nfsrahead callout (issue #158), but it applies to any shimmed binary invoked from a seccomp-sandboxed unit or without the caps the tracer needs. Warn on stderr and exec regardless. The two failure shapes seen in the wild are indistinguishable at this point and both are now non-fatal: a seccomp allow-list (udev) lets the helper run and report its own error over the pipe, while a deny-list kills it with SIGSYS and leaves an empty reply, which previously surfaced as a bare "helper: " with no cause. Refs #158
Reproduces issue #158 without needing udev or an NFS mount: systemd-run with a SystemCallFilter deny-list makes perf_event_open(2) fail for the tracee, which is the same attach failure systemd-udevd's allow-list produces for the sched_process_fork tracepoint. Asserts the shimmed bzip2 still round-trips its payload. Against the previous shim this fails with "shim produced no output -- it exited instead of exec'ing the real binary" (exit 1); with the fix all eight assertions pass. Self-skips where there is no running systemd, so the containerized runner stays green. Listed in run_all_container_tests.sh regardless, to keep that file a complete inventory of the suite.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #159 +/- ##
=======================================
Coverage 67.03% 67.03%
=======================================
Files 20 20
Lines 1805 1805
=======================================
Hits 1210 1210
Misses 595 595 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Regression sweep — all greenHost = local dev box, VM = openSUSE Tumbleweed (kernel 7.2.0), containers run on the VM via podman with Static + unit
Plain e2e on the VM — 14/14 pass
Containerized — both distros succeed
Zero failures in either. The two skips are the pre-existing New test discriminates Same script, two shims:
|
Fixes the second of the two root causes investigated in #158: a shimmed binary
could be prevented from running at all when the tracer failed to attach.
mainstays onfuncBlacklist— the thin-wrapper 0% (cause A in #158,lp/cancel/lpoptions) is accepted as intended semantics and is not touchedhere.
The bug
Tracer.Start()attaches every uprobe throughbpf(2), but thesched_process_forktracepoint goes throughperf_event_open(2). Aseccomp-sandboxed caller commonly permits the former and withholds the latter —
systemd-udevd'sSystemCallFilterlistsbpfbut notperf_event_open— sothat single attach fails while everything else would have worked.
That failure was fatal, and the damage did not stop at coverage:
The real binary never ran. On openSUSE, udev's
nfsraheadcallout(
99-nfs.rules) was replaced by a stub that always failed for as long as theshim stayed installed, silently disabling NFS readahead tuning — and reporting
0% for a binary it never allowed to execute. Any shimmed binary invoked from a
sandboxed unit hits the same path.
The fix
cmd/shim_binary/tracer.go— the fork tracepoint becomes non-fatal. Itonly widens the watched set to child processes, so losing it costs coverage
of forks, never of the process we were asked to trace. Degrades to a debug
log.
cmd/shim_binary/main.go—runWithTracingfails open. If thehelper cannot report ready, warn on stderr and
execthe real binaryanyway. Losing a run's coverage is an acceptable cost of instrumenting a
system; breaking the instrumented program is not.
Both failure shapes seen in the wild are now non-fatal, and they are not
distinguishable at that point in the code: a seccomp allow-list (udev) lets
the helper run and report its own error over the pipe, while a deny-list
kills it with
SIGSYSand leaves an empty reply — which previously surfaced asa bare
helper:with no cause at all.With (1) in place the udev case does not merely survive, it recovers full
coverage: only the fork tracepoint is lost, and a udev callout has no children
to trace.
Verification
Measured on an openSUSE Tumbleweed VM,
nfsraheaddriven through its real udevpath (
udevadm trigger --subsystem-match=bdi --action=add), default udevseccomp filter unmodified:
nfsraheadfunctions tracedfailed with exit code 1)Test
tests/e2e/test_shim_fail_open.shreproduces the attach failure withoutneeding udev or an NFS mount:
systemd-run -p SystemCallFilter=~perf_event_opendenies the same syscall to a shimmed
bzip2and asserts the payload stillround-trips.
FAIL: shim produced no output -- it exited instead of exec'ing the real binary(exit 1)It self-skips where there is no running systemd so the containerized runner
stays green, and is listed in
run_all_container_tests.shregardless to keepthat file a complete inventory.
Regression sweep
go vet ./...clean;go test -race ./...green (host + VM)cpupower, glibc_hwcaps_resolution, duplicate_library_symlink, openssl,
rust_ripgrep, nss_dlopen, pam_dlopen, signal_and_notify_relay, squid,
nginx_dlopen, shim_fail_open
Refs #158