Skip to content

fix(shim): never let a tracer-attach failure stop the real binary from running - #159

Merged
ilmanzo merged 3 commits into
mainfrom
fix/issue-158-shim-fail-open
Aug 31, 2026
Merged

ilmanzo merged 3 commits into
mainfrom
fix/issue-158-shim-fail-open

Conversation

@ilmanzo

@ilmanzo ilmanzo commented Aug 31, 2026

Copy link
Copy Markdown
Owner

Fixes the second of the two root causes investigated in #158: a shimmed binary
could be prevented from running at all when the tracer failed to attach.

main stays on funcBlacklist — the thin-wrapper 0% (cause A in #158,
lp/cancel/lpoptions) is accepted as intended semantics and is not touched
here.

The bug

Tracer.Start() attaches every uprobe through bpf(2), but the
sched_process_fork tracepoint goes through perf_event_open(2). A
seccomp-sandboxed caller commonly permits the former and withholds the latter —
systemd-udevd's SystemCallFilter lists bpf but not perf_event_open — so
that single attach fails while everything else would have worked.

That failure was fatal, and the damage did not stop at coverage:

Tracer.Start() -> error
  -> runHelper writes "ERR:..." to the ready pipe
    -> runWithTracing returns an error
      -> main() os.Exit(1)          <-- before syscall.Exec(realBin)

The real binary never ran. On openSUSE, udev's nfsrahead callout
(99-nfs.rules) was replaced by a stub that always failed for as long as the
shim stayed installed, silently disabling NFS readahead tuning — and reporting
0% for a binary it never allowed to execute. Any shimmed binary invoked from a
sandboxed unit hits the same path.

The fix

  1. cmd/shim_binary/tracer.go — the fork tracepoint becomes non-fatal. It
    only widens the watched set to child processes, so losing it costs coverage
    of forks, never of the process we were asked to trace. Degrades to a debug
    log.
  2. cmd/shim_binary/main.gorunWithTracing fails open. If the
    helper cannot report ready, warn on stderr and exec the real binary
    anyway. Losing a run's coverage is an acceptable cost of instrumenting a
    system; breaking the instrumented program is not.

Both failure shapes seen in the wild are now non-fatal, and they are not
distinguishable at that point in the code: a seccomp allow-list (udev) lets
the helper run and report its own error over the pipe, while a deny-list
kills it with SIGSYS and leaves an empty reply — which previously surfaced as
a bare helper: with no cause at all.

With (1) in place the udev case does not merely survive, it recovers full
coverage: only the fork tracepoint is lost, and a udev callout has no children
to trace.

Verification

Measured on an openSUSE Tumbleweed VM, nfsrahead driven through its real udev
path (udevadm trigger --subsystem-match=bdi --action=add), default udev
seccomp filter unmodified:

before after
nfsrahead functions traced 0/16 — and the binary never ran 10/16, binary runs normally
shim errors in the udev journal 3 (failed with exit code 1) 0

Test

tests/e2e/test_shim_fail_open.sh reproduces the attach failure without
needing udev or an NFS mount: systemd-run -p SystemCallFilter=~perf_event_open
denies the same syscall to a shimmed bzip2 and asserts the payload still
round-trips.

  • against the previous shim: FAIL: shim produced no output -- it exited instead of exec'ing the real binary (exit 1)
  • with this branch: 8/8 assertions pass

It self-skips where there is no running systemd so the containerized runner
stays green, and is listed in run_all_container_tests.sh regardless to keep
that file a complete inventory.

Regression sweep

  • go vet ./... clean; go test -race ./... green (host + VM)
  • all 14 plain e2e tests green on the Tumbleweed VM: bzip2, gzip, gmp,
    cpupower, glibc_hwcaps_resolution, duplicate_library_symlink, openssl,
    rust_ripgrep, nss_dlopen, pam_dlopen, signal_and_notify_relay, squid,
    nginx_dlopen, shim_fail_open
  • containerized suite: see comment below

Refs #158

link.Tracepoint("sched", "sched_process_fork") is the only attach in
Start() that goes through perf_event_open(2) instead of bpf(2). A
seccomp-sandboxed caller commonly permits the latter and withholds the
former -- systemd-udevd's SystemCallFilter lists "bpf" but not
"perf_event_open" -- so this one call fails where every uprobe attaches
fine.

Treating that as fatal cost far more than the tracepoint is worth: the
helper returned an error, runWithTracing propagated it, and main() exited
before ever exec'ing the real binary. A shimmed udev callout was therefore
replaced by a stub that always failed. On openSUSE that silently disabled
NFS readahead tuning for as long as the shim stayed installed, on top of
reporting 0% coverage for a binary that never got the chance to run.

The tracepoint only widens the watched set to child processes, so losing
it costs coverage of forks, never of the process we were asked to trace.
Degrade to a debug log instead.

Refs #158
runWithTracing returned an error whenever the helper failed to report
ready, and main() turns that into os.Exit(1). Because that happens before
syscall.Exec(realBin), an environment that blocks eBPF setup did not just
lose coverage -- it replaced the instrumented program with a stub that
always failed, for as long as the shim stayed installed.

Observed on udev's nfsrahead callout (issue #158), but it applies to any
shimmed binary invoked from a seccomp-sandboxed unit or without the caps
the tracer needs.

Warn on stderr and exec regardless. The two failure shapes seen in the
wild are indistinguishable at this point and both are now non-fatal: a
seccomp allow-list (udev) lets the helper run and report its own error
over the pipe, while a deny-list kills it with SIGSYS and leaves an empty
reply, which previously surfaced as a bare "helper: " with no cause.

Refs #158
Reproduces issue #158 without needing udev or an NFS mount: systemd-run
with a SystemCallFilter deny-list makes perf_event_open(2) fail for the
tracee, which is the same attach failure systemd-udevd's allow-list
produces for the sched_process_fork tracepoint.

Asserts the shimmed bzip2 still round-trips its payload. Against the
previous shim this fails with "shim produced no output -- it exited
instead of exec'ing the real binary" (exit 1); with the fix all eight
assertions pass.

Self-skips where there is no running systemd, so the containerized runner
stays green. Listed in run_all_container_tests.sh regardless, to keep that
file a complete inventory of the suite.
@codecov

codecov Bot commented Aug 31, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 67.03%. Comparing base (5907733) to head (fb6780d).

Files with missing lines Patch % Lines
cmd/shim_binary/main.go 0.00% 4 Missing ⚠️
cmd/shim_binary/tracer.go 0.00% 3 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main     #159   +/-   ##
=======================================
  Coverage   67.03%   67.03%           
=======================================
  Files          20       20           
  Lines        1805     1805           
=======================================
  Hits         1210     1210           
  Misses        595      595           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ilmanzo

ilmanzo commented Aug 31, 2026

Copy link
Copy Markdown
Owner Author

Regression sweep — all green

Host = local dev box, VM = openSUSE Tumbleweed (kernel 7.2.0), containers run on the VM via podman with --privileged --pid=host.

Static + unit

check result
go vet ./... clean
go test -race ./... (host) ok — cmd, cmd/shim_binary, internal/funkutil
go test ./... (VM) ok

Plain e2e on the VM — 14/14 pass

bzip2, gzip, gmp, cpupower, glibc_hwcaps_resolution, duplicate_library_symlink, openssl, rust_ripgrep, nss_dlopen, pam_dlopen, signal_and_notify_relay, squid, nginx_dlopen, shim_fail_open

signal_and_notify_relay and squid are the ones worth calling out — they cover daemon attach, fork-tracking and the sd_notify/PDEATHSIG path, i.e. exactly the behaviour that could plausibly regress from making the sched_process_fork tracepoint non-fatal. Both pass.

Containerized — both distros succeed

runner tests run passed skipped
test_container_tumbleweed.sh 14 11 2
test_container_leap16.sh 14 11 2

Zero failures in either. The two skips are the pre-existing glibc-hwcaps skip and the new shim_fail_open, which self-skips exactly as intended:

=== Prerequisites ===
==> SKIP: no running systemd — cannot build the seccomp sandbox this test needs

New test discriminates

Same script, two shims:

shim result
main @ 5907733 (before) FAIL: shim produced no output -- it exited instead of exec'ing the real binary, exit 1
this branch 8/8 assertions pass, exit 0

@ilmanzo
ilmanzo merged commit 77eaecf into main Aug 31, 2026
3 of 4 checks passed
@ilmanzo
ilmanzo deleted the fix/issue-158-shim-fail-open branch August 31, 2026 09:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant