Repository navigation
Conversation
Both Android arm64 jobs now fail in "Set up Android SDK", before any goffi
code runs:
Warning: Failed to find package 'tools'
Error: The process '/usr/local/lib/android/sdk/cmdline-tools/16.0/bin/sdkmanager' failed with exit code 1
android-actions/setup-android@v3 defaults its `packages` input to
`tools platform-tools` and runs `sdkmanager <pkg>` for each entry. Google's
SDK repository index (repository2-3.xml) no longer contains a `tools`
package, while `platform-tools` and `cmdline-tools` are still listed, so
the `tools` call exits 1. The same breakage is tracked upstream in
android-actions/setup-android#537. The last green run of these jobs on main
was for c8f74c6 on 2026-09-10; the workflow has not changed since, and the
failure is identical on a docs-only PR.
Nothing in this workflow or in scripts/check-android-arm64.sh uses the
`tools` package. Pass `packages: platform-tools`, which keeps everything
the step installed before except the package that no longer exists. The
action still accepts licenses, exports ANDROID_HOME and puts sdkmanager on
PATH, which the following NDK step relies on.
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
A default goffi binary cannot start on Alpine: PT_INTERP names the glibc
loader, which musl systems do not have, and the cgo_import_dynamic
directives name libdl.so.2, libc.so.6 and libpthread.so.0, none of which
musl ships. Its whole POSIX surface lives in one arch-named object,
libc.musl-<arch>.so.1. Both facts are baked into the ELF at link time, so
the libc flavor is a build-time choice.
The goffi_musl tag selects musl flavors of the three directive groups
(internal/dl, internal/syscall, internal/fakecgo) and bakes the musl
loader path into PT_INTERP via //go:cgo_dynamic_linker. That directive is
restricted to cgo-generated code, so musl builds pass
-gcflags=github.com/go-webgpu/goffi/internal/dl=-std. Forgetting the flag
is a compile error naming the directive, not a binary that dies at
startup with a confusing ENOENT.
The glibc libdl imports move out of dl_linux.go into dl_linux_glibc.go so
the RTLD_* constants stay shared by both flavors.
One symbol is dropped from the musl set: pthread_get_stacksize_np is a
Darwin-only API that glibc's lazy PLT tolerates but musl's immediate
binding would reject at load time. Its trampoline is only reachable from
the Darwin thread-entry path, so the linker drops it on Linux. The
fakecgo musl files are generated by gen.go from the same symbol tables as
the glibc ones.
goffi_static wins over goffi_musl; the tag is inert off Linux.
Verification:
- TestMuslDirectiveParity pins glibc/musl symbol-set parity (including
the one intentional exclusion), per-arch SONAMEs and the interpreter.
- TestMuslLinkArtifacts builds linux/{amd64,arm64} probes and checks
PT_INTERP and DT_NEEDED with debug/elf.
- cmd/musl-probe runs against a real musl libc: dlopen/dlsym, float and
integer calls, errno capture, qsort with a Go callback, and a
64-goroutine hammer that makes the runtime create OS threads through
fakecgo's pthread imports. scripts/check-musl.sh runs it inside Alpine
and is wired into CI.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
One CGO_ENABLED=0 binary for linux/amd64 and linux/arm64 that does FFI on both glibc and musl hosts. - Every libc symbol is imported with an empty SONAME (dl_universal.go, errno_universal.go, fakecgo/symbols_universal.go), so the binary has no DT_NEEDED. scripts/build-universal.sh strips PT_INTERP after linking (cmd/goffi-strip-interp), so the kernel loads it directly anywhere; cmd/goffi-audit checks both. - At the top of x_cgo_init, before any libc symbol is touched, the process re-execs itself through the host loader with the host libc preloaded (on glibc also libpthread.so.0 and libdl.so.2, for glibc before 2.34). glibc only binds a main object with PT_INTERP, so there the loader gets a memfd copy with the header restored. Raw syscalls and mmap'd scratch only; setupUniversalTLS gives the first launch a thread pointer. - The guard GOFFI_UNIVERSAL_REEXEC is "<pid>:1", tagged like the recorded GOFFI_UNIVERSAL_EXE/ARGV0, so a child started with exec.Command(os.Args[0]) runs the bridge itself instead of trusting its parent's guard. A child started from the parent's memfd or by the host loader by hand works too. - Where something is preloaded (LD_PRELOAD, /etc/ld.so.preload) the launch is probed in a forked child first. With no recognised loader, or a failed probe, the process carries on as pure Go: iscgo and the libc env hooks are cleared, and LoadLibrary/GetSymbol/CallFunction return ffi.ErrNoHostLibc. - New API: ffi.Available (godoc spells out the three modes), ErrNoHostLibc, HostLoader/HostLibC/HostPreload/LibcKind (from internal/loader), Executable/Argv0. goffi_static wins over goffi_universal. goffi_musl and the default glibc imports are excluded under goffi_universal. CI (universal.yml): build every mode, vet and test; build one probe and run the same binary on the runner, four glibc images (2.31 to current) and Alpine; a respawn probe (cmd/universal-respawn) on all of them; the preload-abort fallback. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
39bdae7 to
0b00026
Compare
kolkov
left a comment
There was a problem hiding this comment.
Thanks for this, and for answering the two questions from #76 so concretely. Removing the exec.Command footgun with the pid-tagged guard is better than the wrapper I suggested.
I reviewed only the last commit and tested it on real hosts rather than just reading it.
What works (verified)
WSL Ubuntu 22.04 (glibc 2.35), Go 1.27.1, plus an Alpine 3.24 chroot:
- One binary from
scripts/build-universal.shmakes FFI calls on glibc and on musl (getpidthrough goffi matchesos.Getpid, andLibcKindreports correctly). cmd/universal-probe→UNIVERSAL-PROBE-OKandcmd/universal-respawn→RESPAWN-PROBE-OKon both.- No loader at all (an empty chroot with only
/proc): the bridge prints its notice, the process keeps running,Available() == false, andLoadLibraryreturnsErrNoHostLibc. Theiscgo = falsefallback holds up. go test ./...passes in the default build, with-tags goffi_universal, and withCGO_ENABLED=1.
The core idea holds up. Below are the problems I found, roughly in order of severity.
Must fix: argv/env corruption in the bridge
reexecUniversal reads /proc/self/cmdline and /proc/self/environ into fixed 256 KiB buffers and builds argv/envp in fixed 4096-pointer arrays. When any of these overflow, the bridge truncates silently and still calls execve:
$ ./uprobe args $(seq 1 5000) # universal build
argc=4092 ... # ~900 arguments dropped, no error
$ ./uprobe-default args $(seq 1 5000) # default build
argc=5002
$ ./uprobe args "$A" "$A" "$A" "${A}END" # 4 × 100 KB
argc=5 last="aaaa…aaaaadjustdefers" # one argument lost, last one cut off,
# and bytes from some other mapping appended
# (which bytes varies from build to build)
$ env V1=<10KB> … V30=<10KB> BIGVAR=lastvalue ./uprobe env
BIGVAR len=0 # default build: len=9
The appended garbage in the second case shows that the last argument is no longer NUL-terminated inside the buffer, so execve reads past it.
This is reachable in practice, not only in a contrived test. ARG_MAX is 2 MiB here, and plain xargs with default settings (128 KiB per invocation) already crosses the 4096-pointer limit with short arguments:
$ seq 1 30000 | xargs ./uprobe args # universal
argc=4092 ... # xargs passed 23 694 arguments
$ seq 1 30000 | xargs ./uprobe-default args # default build
argc=23694 ...
For a file manager like f4 that is started with many file arguments, it's data loss.
Please size the buffers from what's actually there (read in a loop and grow with a fresh mmap, and count entries before filling the pointer arrays). Where something still can't fit, don't exec: fall back to the no-FFI path. A wrong argv is much worse than no FFI.
Should fix: what leaks into child processes
1. The memfd is inherited by every child. It's created without MFD_CLOEXEC so the loader can open /proc/self/fd/<n>. That's right for the bridge's own execve, but it stays open in the running program afterwards:
self fd 3 -> /memfd:goffi (deleted)
--- child (sh) view:
lrwx------ ... 3 -> /memfd:goffi (deleted)
Every process the program starts (shells, helpers, anything a C library spawns) holds a full in-memory copy of the binary for as long as it lives. Setting FD_CLOEXEC on it once the runtime is up keeps exec.Command(os.Args[0]) working, because the kernel resolves /proc/self/fd/<n> before CLOEXEC descriptors are closed, and stops the leak.
2. GOFFI_UNIVERSAL_* variables are inherited by every child, goffi or not (minor):
GOFFI_UNIVERSAL_ARGV0=612:./uprobe
GOFFI_UNIVERSAL_REEXEC=612:1
GOFFI_UNIVERSAL_EXE=612:/home/andy/uprobe/uprobe
The pid tagging keeps this correct, so it isn't a bug. They do show up in os.Environ() and in the environment of unrelated child processes, though. They can't simply be unset at init: a child started from the parent's memfd (os.Args[0] = /proc/self/fd/<n> on glibc) reads the parent's GOFFI_UNIVERSAL_EXE by ppid, both in the bridge and in ffi.Executable(). So I'd settle for documenting them in PROFILE_U.md, unless you see a cheap way to scope them to goffi children.
3. The guard could also check /proc/self/exe. After a successful re-exec, /proc/self/exe is always the loader. If the guard required both the pid tag and that, a pid-reuse false positive (an ancestor's guard reaching a later process with the same pid, more plausible in containers with a small pid_max) couldn't skip the bridge and crash on the first unbound malloc. The check already exists further down (sameBase(realExeC, …)).
Should fix: failures that kill the process instead of falling back
The fallback covers failures up to and including the execve: no loader file, no scratch memory or unreadable /proc, execve itself failing, and a preloaded library failing in the probe. Any failure after a successful execve of the loader kills the process before main, and the docs promise "starts anywhere". Cases I'd expect in the wild:
- glibc < 2.30, which
docs/PROFILE_U.mddocuments. But that range includes RHEL/Rocky/Alma 8 (glibc 2.28, supported until 2029) and Amazon Linux 2 (2.26), not only Ubuntu 18.04. For those users the binary doesn't degrade, it dies.LD_PRELOADin the re-exec environment works on every glibc, and the bridge already rewrites the environment, so the re-executed process could restore the user's originalLD_PRELOAD(or its absence) before anything else runs. Alternatively, detect the old loader and take the fallback path. - A file at the loader path that isn't a working glibc loader, for example NixOS's stub at
/lib64/ld-linux-x86-64.so.2(environment.stub-ld). vm.memfd_noexec=2(kernel 6.3+): perDocumentation/userspace-api/mfd_noexec.rst, "memfd_create()withoutMFD_NOEXEC_SEALwill be rejected", so both theMFD_EXECattempt and theflags=0retry fail. The bridge then hands the glibc loader the on-disk image, which its own comment says won't bind. (=1is fine:MFD_EXECis still honoured there.)
A cheap way to cover all of them is to run the existing probeHostLoader on every launch, not only when something is preloaded, and write the cost down. The probe child exits non-zero in each of the cases above, so the process would fall back instead of dying. I'd rather pay a fork+exec at startup than ship a binary that sometimes refuses to start.
Rebase: the set*id hooks from #92
#92 (merged) added fakecgo hooks for syscall.Setuid, Setgid, Setgroups and the rest (internal/fakecgo/linux.go, ported from purego#426). Without them, those calls panic once fakecgo sets iscgo. The hooks are installed statically, and the file builds under goffi_universal too. So besides resolving the conflicts in gen.go, go_linux_{amd64,arm64}.go and symbols_linux.go, the rebase needs two things:
symbols_universal.gomust import what #92 added tosymbols_linux.go:__errno_locationand nine set*id/setgroupsfunctions. Please regenerate it rather than merging the generated files by hand.- The
hostlibc.Missingfallback has to clear these hooks, the same waydropLibcEnvHooksclears the setenv ones.syscall.Setuidcalls the hook throughcgocallerwhenever it is non-nil (syscall/syscall_linux.go), andruntime.cgocallthrowscgocall unavailableonceiscgois false. With the hooks nil,syscallusesAllThreadsSyscall, which works in a process without cgo.
Running a setuid check as root in the universal jobs (glibc, Alpine and the no-loader case) would cover both. #92's TestSetuidEtc in internal/fakecgo shows the approach.
Needs a decision: arm64
setup_universal_tls_arm64.s says arm64 "has not been run-tested", and PROFILE_U.md says it's cross-compile-verified only. Yet the PR description and CHANGELOG present linux/arm64 as supported without that caveat. ci.yml has run tests on ubuntu-24.04-arm since #92, and those runners are free for public repos. Could universal.yml run the glibc and Alpine jobs there too? Until it does, please mark arm64 as experimental in the CHANGELOG and README row as well.
Public API
The PR adds eight exported identifiers to ffi. Available() and ErrNoHostLibc belong there: Available() is useful for goffi_static users as well. The rest (HostLoader, HostLibC, HostPreload, LibcKind, Executable, Argv0) only make sense for universal builds on Linux, but they'd sit in the core package's godoc on every platform. HostPreload() returns a space-separated string and LibcKind() a string where a type would do. I'd prefer moving these into a subpackage (github.com/go-webgpu/goffi/universal or similar), where the API can evolve without touching the core package. Once these are in a release we're stuck with them, so it's easier to get this right now.
Build ergonomics
The binary only behaves as documented after goffi-strip-interp. A plain go build -tags goffi_universal keeps the glibc PT_INTERP. It works on a glibc dev machine and fails on Alpine with No such file or directory. I checked both. That is precisely the "works on my machine" failure mode. Please state it at the top of PROFILE_U.md and in the README row, and show downstream users how to run the tool from the module (go run github.com/go-webgpu/goffi/cmd/goffi-strip-interp@<version>), since they won't have our scripts/.
CI
universal.ymlisn't part ofCI Success, so none of it can become a required check as it stands. Please either fold the jobs intoci.ymlor add a final success job we can require.on: push: branches: ["**"]pluspull_requestruns everything twice on every PR push.- It pins Go
1.26.xonly, and the comment says that matchesci.yml, butci.ymltests 1.25 (the go.mod minimum) and, since #92, Go stable as well. - The comment above
go vet -unsafeptr=falserefers toffi/callback_pointer.goand its//go:nocheckptrcontract, and that file doesn't exist upstream (probably left over from the fork).
Questions
- setuid / file capabilities: re-exec'ing through
ld.somay change whatAT_SECUREand the inherited credentials look like compared with a normal launch. I haven't tested this. Has it been considered? If it isn't supported, the bridge could checkAT_SECUREin auxv and take the no-FFI path, and the docs should say so. unxed/static-everywherehas no license file (GitHub reports none), andNOTICEsays this build ports its concept. Since you're the author of both, a one-line confirmation here that you contribute it under goffi's license (or adding a license there) would keep the attribution clean.
License headers
goffi is MIT-licensed (LICENSE). Apache-2.0 is only right for files derived from purego. NOTICE covers those, and they carry SPDX-FileCopyrightText: 2022 The Ebitengine Authors. symbols_universal.go (generator-style, Ebitengine header) is fine. The ~30 other new files (reexec_*, setup_universal_tls*, internal/loader, internal/hostlibc, ffi/available.go, ffi/selfinfo.go, ffi/hostinfo.go, the cmd/ tools, tests, scripts/build-universal.sh) are original code and should be:
// SPDX-License-Identifier: MIT
// SPDX-FileCopyrightText: 2026 Andrey Kolkov and GoGPU ContributorsThe Apache pattern came from our own files on main, so that part is on us, and we're fixing main separately. Mixed headers in an MIT project trip corporate license scanners, so it's worth getting right in new code.
Order: this needs #87 settled first (see my review there: the musl part may shrink to a linker flag, which changes the build-tag exclusions here). The argv/env truncation is the one item I'd treat as a hard blocker regardless. Everything else is open to discussion.
Step 4 of the split agreed in #76. Depends on #87 (
goffi_musl). The branch is stacked on it, so review only the last commit.-tags goffi_universalbuilds oneCGO_ENABLED=0binary for linux/amd64 and linux/arm64 that does FFI on both glibc and musl hosts.How it works
DT_NEEDED.scripts/build-universal.shthen removesPT_INTERP, andcmd/goffi-auditchecks both. The kernel can load the binary directly on any distro.x_cgo_init, before any libc symbol is used, the process re-execs itself through the host's dynamic loader with the host libc preloaded. On glibc it also preloadslibpthread.so.0andlibdl.so.2, which older glibc needs. glibc's loader only binds a main object that hasPT_INTERP, so on glibc it gets an in-memory copy with that header restored. This code uses only raw syscalls and mmap'd scratch memory.LD_PRELOADor/etc/ld.so.preloadkills the re-executed process, the program keeps running as pure Go.LoadLibrary,GetSymbolandCallFunctionthen returnffi.ErrNoHostLibc. When something is preloaded, the launch is first tried in a forked child.The
exec.Commandquestion from #76. I removed the footgun instead of wrapping it:GOFFI_UNIVERSAL_REEXECis now<pid>:1, tagged the same way asGOFFI_UNIVERSAL_EXEandGOFFI_UNIVERSAL_ARGV0. The re-executed process keeps its pid, so it finds its own guard. A child has a new pid, so it runs the bridge itself.cmd/universal-respawnstarts itself viaos.Args[0], viaffi.Executable(), and as a grandchild. Each child must callgetpidandstrlenthrough goffi and report the on-diskffi.Executable()./proc/self/fd/<n>image withloader cannot load itself. With the pid-tagged guard, that recipe is no longer needed anyway.ffi.Available(). The godoc now spells out the three modes:goffi_static;goffi_universal;This PR adds
Availableto upstream, along with:ErrNoHostLibc;HostLoader,HostLibC,HostPreloadandLibcKind(frominternal/loader);ExecutableandArgv0, which return whatos.Executable()andos.Args[0]gave before the re-exec.Tags:
goffi_statictakes precedence overgoffi_universal. Undergoffi_universal, the glibc andgoffi_muslimport files are excluded.CI: new
universal.yml. It builds every mode, then runs vet and tests. It builds one probe binary and runs that same binary on glibc and musl, runs the respawn probe everywhere, and runs the preload-abort fallback jobs. Everything passes. Inci.ymlonly the Android arm64 jobs fail, in "Set up Android SDK", which #85 fixes.Docs:
docs/PROFILE_U.md, README, CHANGELOG, NOTICE (attribution tounxed/static-everywhere).🤖 Generated with Claude Code