spanner_omni_crash_report.tar.gz
Bug report
Summary
After Docker Desktop auto-updated from 4.86.0 to 4.87.0 on an Apple M1 Max, a container
workload that had been stable for six days started crashing continuously: a process inside
the container died 120 times in 24 minutes with a mix of SIGSEGV / SIGABRT / SIGBUS / SIGTRAP
and non-deterministic fault addresses.
Downgrading to 4.86.0 - with every other variable held constant (same image digest, same
compose file, same host, same test suite) - eliminated the crashes completely (0 in 25
minutes of the same load).
The only component that differs between the two Docker Desktop versions on this machine is
the VM's Linux kernel (and the VM root filesystem image). The Docker Engine binary is
identical. The kernel change is not mentioned anywhere in the release notes for either
version, and it moved from an LTS series (6.12) to a new series (7.0).
We are not yet able to say whether the defect is in kernel 7.0.12 itself or whether the new
kernel exposes a latent bug in the workload. We are filing this with Docker because the
undocumented kernel change is what changed on our side, and because the delivery mechanism
(silent auto-update) made it hard to diagnose. A separate report is being filed in parallel with
the vendor of the affected workload.
Environment
Host
Machine : Apple M1 Max (arm64)
macOS : Darwin 25.6.0
Host disk free : 595 GB
Docker Desktop (bad)
Version : 4.87.0 (build 236836)
Installed : 2026-08-18T06:22:47Z (auto-update; restarted as 4.87.0 at 06:22:56Z)
Docker Engine : 29.7.2 (build a7dcaa6)
VM kernel : 7.0.12-linuxkit
Docker Desktop (good)
Version : 4.86.0 (build 236216)
In use since : 2026-08-11T23:21:37Z
Docker Engine : 29.7.2 (build a7dcaa6) <-- identical binary
VM kernel : 6.12.76-linuxkit
VM configuration (identical in both versions)
Virtualization : Apple Virtualization.framework
(UseLibkrun=false, UseVirtualizationFramework=true,
UseVirtualizationFrameworkVirtioFS=true,
UseVirtualizationFrameworkRosetta=true)
Network type : gvisor
CPUs / Memory : 10 / 33024 MiB
Kernel cmdline : init=/initd loglevel=1 root=/dev/vdb rootfstype=erofs ro
vsyscall=emulate panic=0 eth0.dhcp eth1.dhcp
linuxkit.unified_cgroup_hierarchy=1 console=hvc0
virtio_net.disable_csum=1 slub_min_order=2
page_reporting.page_reporting_order=2
vpnkit.connect=connect://2/1999
Note: the kernel command line is byte-for-byte the same under both versions, so the
difference is in the kernel implementation, not in how it is configured.
What actually differs between 4.86.0 and 4.87.0 on this machine
Both application bundles were present locally (4.87.0 was still staged in
~/Library/Application Support/com.docker.install/in_progress/Docker.app after the
downgrade), so they could be compared directly:
Component 4.86.0 (236216) 4.87.0 (236836)
Contents/Resources/linuxkit/kernel 6.12.76-linuxkit 7.0.12-linuxkit
size 36,839,432 bytes 39,069,704 bytes
sha256 (prefix) dca24feebdd26186... c970844e055ea99a...
Contents/Resources/linuxkit/desktop.img 665,550,848 bytes 681,357,312 bytes
Contents/Resources/bin/docker 29.7.2 (a7dcaa6) 29.7.2 (a7dcaa6) [same]
The kernel actually booted by the VM is corroborated by the VM console logs:
~/Library/Containers/com.docker.docker/Data/log/vm/
console.log.20260818-152254.709 -> "Linux version 6.12.76-linuxkit" (rotated at update time)
console.log -> "Linux version 7.0.12-linuxkit"
Neither the 4.86.0 nor the 4.87.0 release notes mention a kernel version or a kernel upgrade.
The affected workload
Image : us-docker.pkg.dev/spanner-omni/images/spanner-omni:2026.r1-beta.3
@sha256:de8a97708909121630d6ad08757bf9e2c721bb70fdbbe888944843748611b3cf
Command : start-single-server
Also reproduced with 2026.r1-beta.4
@sha256:f3ce76b98dccef8ea53a32e17fdc794d9b3a1e509c8c75e5619cff518157d1bf
This is Google's "Spanner Omni" (Preview), a self-managed Spanner distribution. Inside the
container, a supervisor process (/google/spanner/bin/spanner, under tini) starts three
children of a large C++ binary (spanner_server, 660 MB, ~1,500 memory mappings, heavily
threaded), named "server", "base_services" and "zone_services".
It is the child processes that crash. The supervisor restarts them in about 0.1 s.
Symptoms under kernel 7.0.12-linuxkit
Under load (a 25-minute end-to-end test suite):
Window : 2026-08-18T07:28:21Z - 07:52:42Z (24 minutes)
Crashes : 120
Mean interval : 12.3 s (min 0.2 s, max 160.6 s)
Signals : segmentation fault 90, aborted 25, bus error 4, trace/breakpoint trap 1
Crashing children : server 115, base_services 4, zone_services 1
Representative supervisor output:
07:28:21 Server zone_services has stopped: failed to run spanner_server:
failed to run spanner_server: signal: segmentation fault
07:28:23 Server server has stopped: ... signal: segmentation fault
07:29:57 Server server has stopped: ... signal: bus error
07:31:02 Server server has stopped: ... signal: aborted
07:43:58 Server server has stopped: ... signal: trace/breakpoint trap
At startup, with a freshly created volume and no clients connected at all:
Container started 08:10:35Z -> 21 crashes between 08:10:37Z and 08:12:02Z
"Spanner is ready" only reached at 08:12:22Z (105 s)
An isolated container with no published ports and no clients whatsoever also crashed:
Started 08:14:21Z, "Spanner is ready" 08:14:27Z
08:14:39Z aborted / 08:14:50Z bus error / 08:15:02Z segfault / 08:15:14Z segfault
(intervals of 11.62 s, 11.61 s, 11.74 s), then quiet while idle
So the crashes occur whenever the process has work to do (its own initialisation is enough)
and are not caused by client traffic.
Crash evidence: this is memory corruption, and it is non-deterministic
The C++ binary writes its own crash dumps to /spanner/logs//STDOUT.*. Four crashes
from one idle container were collected. All four differ in fault address, stack, thread and
CPU:
instance crash time signal reported signal in fault address si_code pid/tid cpu
started (supervisor) by supervisor the dump
1 08:14:21Z 08:14:39Z aborted SIGSEGV 0x1fffffff0 1 (SEGV_MAPERR) 21/787 4
2 08:14:39Z 08:14:50Z bus error SIGBUS 0x61767265746e69 1 1086/1441 1
3 08:14:50Z 08:15:02Z segfault SIGSEGV 0x7330f302fa60 2 (SEGV_ACCERR) 1559/1974 7
4 08:15:02Z 08:15:14Z segfault SIGSEGV 0x132808080 1 (SEGV_MAPERR) 2037/2051 4
(Dump files are named after the instance's start time, so each row pairs one instance's dump
with the death the supervisor logged for it. In row 1 the supervisor reported "aborted" while
the dump records SIGSEGV; the crash handler re-raises the signal, so the exit status seen by
the parent does not always match the original one.)
Crash #2 is the most informative. The fault address decodes as ASCII text:
0x61767265746e69 -> little-endian bytes 69 6e 74 65 72 76 61 00 -> "interva\0"
and the reported PC is that same address (the stack trace begins with
"trace=61767265746e69,..."), i.e. control was transferred into string data. Crash #3 has the
same shape (PC == fault address). In the dump quoted below the loaded binary occupies
0xaaaadc200000-0xaaaafd40fbc0, so these fault addresses are far outside any mapping.
All stacks are ~24 frames deep and end in start_thread, i.e. the crashes happen on
background threads, not on request-handling threads. Symbols are stripped and the dumps
reference an internal Google symbol server, so we cannot resolve function names.
Example dump excerpt:
*** SIGSEGV (@0xe8075ff0e8075), si_code=1, see go/stacktraces#s15 received by
PID 252 (TID 337) on cpu 3; stack trace: ***
PC: @ 0xaaaafafa81ec (unknown) (unknown)
...
@ 0xffff8c67dd4c 288 start_thread
E0818 08:10:38 coredump_hook.cc:278] RAW: Remote crash gathering disabled
E0818 08:10:38 process_state.cc:704] RAW: Raising signal 11 with default behavior
A/B result: the kernel is the trigger
Everything except the Docker Desktop version was held constant (same image digest, same
compose.yaml, same host, same 25-minute test suite):
Metric kernel 7.0.12 kernel 6.12.76
spanner_server crashes 120 (24 min) 0 (25 min)
time to "Spanner is ready" 105 s 9 s
gRPC failures seen by clients 4 0
"connection reset by peer" to the DB 8 0
end-to-end test failures 4 0
The crash-free run under 6.12.76 was measured with two independent methods that agreed:
polling every 20 s (12 container generations tracked, since the suite recreates the
container between phases) and an event-driven collector that attached "docker logs -f" on
each "start" event so short-lived containers could not be missed.
What we ruled out
- Workload version : both 2026.r1-beta.3 and 2026.r1-beta.4 crash identically
- Corrupted persistent state : the data volume is recreated on every run
- Out of memory : OOMKilled=false, no memory limit, 1.26 GiB used of 31.5 GiB
- Disk exhaustion : 595 GB free on the host, 44 GB free inside the VM
- /dev/shm exhaustion : 64 MB, 0% used, and spanner_server has zero /dev/shm mappings
- Emulation : the image is arm64 and the host is arm64 (no qemu/Rosetta path)
- Client-triggered : an idle container with no published ports also crashes
- Guest kernel errors : console logs under both kernels contain no
BUG:/WARNING:/Oops/call trace/corruption messages
Other containers on the same VM are unaffected
21 other containers were running on the same VM throughout, all with RestartCount=0 and no
OOM kills: three BigQuery emulators, Firestore emulator, fake-gcs-server, Envoy 1.32.1,
Jaeger, Cloud Spanner Emulator 1.5.55, an OpenTelemetry collector, and several Go services.
Only the heaviest process - a 660 MB C++ binary with ~1,500 mappings and many threads -
is affected. We mention this both as a data point and because it means we cannot show
VM-wide corruption.
Why this was hard to detect
The crash is invisible from Docker's point of view. The in-container supervisor restarts the
child within ~0.1 s, so:
docker inspect -> RestartCount=0, State.Status=running, Health=healthy
docker ps -> "Up 23 minutes (healthy)"
Nothing suggests a problem unless you read the container's stdout. We only found it by
tracing gRPC "DeadlineExceeded" errors in application logs back to
"error reading server preface: read tcp [::1]:58770->[::1]:15000: read: connection reset by
peer" - i.e. the database process disappearing mid-request.
Reproduction
- Docker Desktop 4.87.0 on Apple Silicon (kernel 7.0.12-linuxkit).
- docker run -d --rm --name omni-repro
us-docker.pkg.dev/spanner-omni/images/spanner-omni:2026.r1-beta.4
start-single-server
- Wait ~60 s, then:
docker logs omni-repro | grep "has stopped"
Expected on 7.0.12 : several "signal: segmentation fault" / "bus error" / "aborted" lines
Expected on 6.12.76: no output
- Full dumps: docker cp omni-repro:/spanner/logs ./logs
No ports, volumes or client traffic are required.
Impact
- A test suite that had been green for six days produced random failures after the update.
- Diagnosis cost roughly a full day, because Docker reports the container as healthy.
- The kernel change that caused it is not in the release notes, so there was no reason to
suspect Docker Desktop at first; time was spent investigating the workload's own
versions instead.
Questions and requests
- Was the VM kernel intentionally moved from 6.12.76 (LTS) to 7.0.12 in 4.87.0, and if so
why is it not in the release notes? Kernel changes are visible to workloads and should
be documented.
- Are there known issues with 7.0.12-linuxkit and memory-intensive workloads? In
particular, is free page reporting (page_reporting.page_reporting_order=2, which is
passed on the kernel command line) known to be safe on this kernel in combination with
Apple Virtualization.framework?
- Is there any supported way to pin or select the VM kernel, so users can A/B test or work
around a kernel regression without downgrading the whole application?
- If you would like additional data - full crash dumps, diagnostics bundles from both
versions, or a run with specific kernel parameters changed - we can reproduce on demand.
Current workaround
Staying on Docker Desktop 4.86.0 with automatic updates disabled
(DisableUpdate=true, AutoDownloadUpdates=false).
Attachments
Prepared as spanner_omni_crash_report.tar.gz (2.1 MB):
README.txt what is in the archive and how it was collected
01_crash_summary.txt the four crash signatures side by side, incl. the ASCII pointer
02_supervisor_log.txt the supervisor's record of the child deaths
03_environment.txt host / Docker / kernel / image digests / container layout
04_docker_version_diff.txt the measured 4.86.0 vs 4.87.0 comparison (kernel, desktop.img,
engine binary) and the update timeline from the install log
05_ab_result.txt the A/B measurement and how the 0-crash result was verified
crash_stacks/ the four STDOUT files containing a crash dump
dumps/ the complete, unmodified /spanner/logs tree from the container
Can also be provided on request:
- VM console logs from both kernels
(~/Library/Containers/com.docker.docker/Data/log/vm/console.log*)
- a Docker Desktop diagnostics bundle (4.87.0 would have to be reinstalled to produce one)
Platform
macOS
Version information
Client:
Version: 29.7.2
API version: 1.55
Go version: go1.26.5
Git commit: a7dcaa6
Built: Wed Aug 5 18:27:50 2026
OS/Arch: darwin/arm64
Context: desktop-linux
Server: Docker Desktop 4.86.0 (236216)
Engine:
Version: 29.7.2
API version: 1.55 (minimum version 1.40)
Go version: go1.26.5
Git commit: 6a43e3d
Built: Wed Aug 5 18:28:35 2026
OS/Arch: linux/arm64
Experimental: false
containerd:
Version: v2.2.5
GitCommit: e53c7c1516c3b2bff98eb76f1f4117477e6f4e66
runc:
Version: 1.3.6
GitCommit: v1.3.6-0-g491b69ba
docker-init:
Version: 0.19.0
GitCommit: de40ad0
Client:
Version: 29.7.2
Context: desktop-linux
Debug Mode: false
Plugins:
agent: Docker AI Agent Runner (Docker Inc.)
Version: v1.119.0
Path: /Users/yoshikishibata/.docker/cli-plugins/docker-agent
ai: Docker AI Agent - Ask Gordon (Docker Inc.)
Version: v1.30.0
Path: /Users/yoshikishibata/.docker/cli-plugins/docker-ai
buildx: Docker Buildx (Docker Inc.)
Version: v0.36.0-desktop.1
Path: /Users/yoshikishibata/.docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v5.3.1
Path: /Users/yoshikishibata/.docker/cli-plugins/docker-compose
debug: Get a shell into any image or container (Docker Inc.)
Version: 0.0.47
Path: /Users/yoshikishibata/.docker/cli-plugins/docker-debug
desktop: Docker Desktop commands (Docker Inc.)
Version: v0.4.3
Path: /Users/yoshikishibata/.docker/cli-plugins/docker-desktop
dhi: CLI for managing Docker Hardened Images (Docker Inc.)
Version: v0.0.7
Path: /Users/yoshikishibata/.docker/cli-plugins/docker-dhi
extension: Manages Docker extensions (Docker Inc.)
Version: v0.2.31
Path: /Users/yoshikishibata/.docker/cli-plugins/docker-extension
init: Creates Docker-related starter files for your project (Docker Inc.)
Version: v1.4.0
Path: /Users/yoshikishibata/.docker/cli-plugins/docker-init
mcp: Docker MCP Plugin (Docker Inc.)
Version: v0.43.3
Path: /Users/yoshikishibata/.docker/cli-plugins/docker-mcp
model: Docker Model Runner (Docker Inc.)
Version: v1.2.6
Path: /Users/yoshikishibata/.docker/cli-plugins/docker-model
offload: Docker Offload (Docker Inc.)
Version: v0.6.9
Path: /Users/yoshikishibata/.docker/cli-plugins/docker-offload
pass: Docker Pass Secrets Manager Plugin (beta) (Docker Inc.)
Version: v0.2.0
Path: /Users/yoshikishibata/.docker/cli-plugins/docker-pass
sandbox: "docker sandbox" is deprecated, use Docker Sandboxes instead (Docker Inc.)
Version: v0.13.0
Path: /Users/yoshikishibata/.docker/cli-plugins/docker-sandbox
scout: Docker Scout (Docker Inc.)
Version: v1.24.0
Path: /Users/yoshikishibata/.docker/cli-plugins/docker-scout
Server:
Containers: 29
Running: 22
Paused: 0
Stopped: 7
Images: 43
Server Version: 29.7.2
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
CDI spec directories:
/etc/cdi
/var/run/cdi
Discovered Devices:
cdi: docker.com/gpu=webgpu
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: e53c7c1516c3b2bff98eb76f1f4117477e6f4e66
runc version: v1.3.6-0-g491b69ba
init version: de40ad0
Security Options:
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.12.76-linuxkit
Operating System: Docker Desktop
OSType: linux
Architecture: aarch64
CPUs: 10
Total Memory: 31.54GiB
Name: docker-desktop
ID: ed5c4638-42d5-47dc-9dc4-d6196b14bae9
Docker Root Dir: /var/lib/docker
Debug Mode: false
HTTP Proxy: http.docker.internal:3128
HTTPS Proxy: http.docker.internal:3128
No Proxy: hubproxy.docker.internal
Labels:
com.docker.desktop.address=unix:///Users/yoshikishibata/Library/Containers/com.docker.docker/Data/docker-cli.sock
Experimental: false
Insecure Registries:
hubproxy.docker.internal:5555
::1/128
127.0.0.0/8
Live Restore Enabled: false
Firewall Backend: iptablesDiagnostics ID
No response
spanner_omni_crash_report.tar.gz
Bug report
Summary
After Docker Desktop auto-updated from 4.86.0 to 4.87.0 on an Apple M1 Max, a container
workload that had been stable for six days started crashing continuously: a process inside
the container died 120 times in 24 minutes with a mix of SIGSEGV / SIGABRT / SIGBUS / SIGTRAP
and non-deterministic fault addresses.
Downgrading to 4.86.0 - with every other variable held constant (same image digest, same
compose file, same host, same test suite) - eliminated the crashes completely (0 in 25
minutes of the same load).
The only component that differs between the two Docker Desktop versions on this machine is
the VM's Linux kernel (and the VM root filesystem image). The Docker Engine binary is
identical. The kernel change is not mentioned anywhere in the release notes for either
version, and it moved from an LTS series (6.12) to a new series (7.0).
We are not yet able to say whether the defect is in kernel 7.0.12 itself or whether the new
kernel exposes a latent bug in the workload. We are filing this with Docker because the
undocumented kernel change is what changed on our side, and because the delivery mechanism
(silent auto-update) made it hard to diagnose. A separate report is being filed in parallel with
the vendor of the affected workload.
Environment
Host
Machine : Apple M1 Max (arm64)
macOS : Darwin 25.6.0
Host disk free : 595 GB
Docker Desktop (bad)
Version : 4.87.0 (build 236836)
Installed : 2026-08-18T06:22:47Z (auto-update; restarted as 4.87.0 at 06:22:56Z)
Docker Engine : 29.7.2 (build a7dcaa6)
VM kernel : 7.0.12-linuxkit
Docker Desktop (good)
Version : 4.86.0 (build 236216)
In use since : 2026-08-11T23:21:37Z
Docker Engine : 29.7.2 (build a7dcaa6) <-- identical binary
VM kernel : 6.12.76-linuxkit
VM configuration (identical in both versions)
Virtualization : Apple Virtualization.framework
(UseLibkrun=false, UseVirtualizationFramework=true,
UseVirtualizationFrameworkVirtioFS=true,
UseVirtualizationFrameworkRosetta=true)
Network type : gvisor
CPUs / Memory : 10 / 33024 MiB
Kernel cmdline : init=/initd loglevel=1 root=/dev/vdb rootfstype=erofs ro
vsyscall=emulate panic=0 eth0.dhcp eth1.dhcp
linuxkit.unified_cgroup_hierarchy=1 console=hvc0
virtio_net.disable_csum=1 slub_min_order=2
page_reporting.page_reporting_order=2
vpnkit.connect=connect://2/1999
Note: the kernel command line is byte-for-byte the same under both versions, so the
difference is in the kernel implementation, not in how it is configured.
What actually differs between 4.86.0 and 4.87.0 on this machine
Both application bundles were present locally (4.87.0 was still staged in
~/Library/Application Support/com.docker.install/in_progress/Docker.app after the
downgrade), so they could be compared directly:
Component 4.86.0 (236216) 4.87.0 (236836)
Contents/Resources/linuxkit/kernel 6.12.76-linuxkit 7.0.12-linuxkit
size 36,839,432 bytes 39,069,704 bytes
sha256 (prefix) dca24feebdd26186... c970844e055ea99a...
Contents/Resources/linuxkit/desktop.img 665,550,848 bytes 681,357,312 bytes
Contents/Resources/bin/docker 29.7.2 (a7dcaa6) 29.7.2 (a7dcaa6) [same]
The kernel actually booted by the VM is corroborated by the VM console logs:
~/Library/Containers/com.docker.docker/Data/log/vm/
console.log.20260818-152254.709 -> "Linux version 6.12.76-linuxkit" (rotated at update time)
console.log -> "Linux version 7.0.12-linuxkit"
Neither the 4.86.0 nor the 4.87.0 release notes mention a kernel version or a kernel upgrade.
The affected workload
Image : us-docker.pkg.dev/spanner-omni/images/spanner-omni:2026.r1-beta.3
@sha256:de8a97708909121630d6ad08757bf9e2c721bb70fdbbe888944843748611b3cf
Command : start-single-server
Also reproduced with 2026.r1-beta.4
@sha256:f3ce76b98dccef8ea53a32e17fdc794d9b3a1e509c8c75e5619cff518157d1bf
This is Google's "Spanner Omni" (Preview), a self-managed Spanner distribution. Inside the
container, a supervisor process (/google/spanner/bin/spanner, under tini) starts three
children of a large C++ binary (spanner_server, 660 MB, ~1,500 memory mappings, heavily
threaded), named "server", "base_services" and "zone_services".
It is the child processes that crash. The supervisor restarts them in about 0.1 s.
Symptoms under kernel 7.0.12-linuxkit
Under load (a 25-minute end-to-end test suite):
Window : 2026-08-18T07:28:21Z - 07:52:42Z (24 minutes)
Crashes : 120
Mean interval : 12.3 s (min 0.2 s, max 160.6 s)
Signals : segmentation fault 90, aborted 25, bus error 4, trace/breakpoint trap 1
Crashing children : server 115, base_services 4, zone_services 1
Representative supervisor output:
07:28:21 Server zone_services has stopped: failed to run spanner_server:
failed to run spanner_server: signal: segmentation fault
07:28:23 Server server has stopped: ... signal: segmentation fault
07:29:57 Server server has stopped: ... signal: bus error
07:31:02 Server server has stopped: ... signal: aborted
07:43:58 Server server has stopped: ... signal: trace/breakpoint trap
At startup, with a freshly created volume and no clients connected at all:
Container started 08:10:35Z -> 21 crashes between 08:10:37Z and 08:12:02Z
"Spanner is ready" only reached at 08:12:22Z (105 s)
An isolated container with no published ports and no clients whatsoever also crashed:
Started 08:14:21Z, "Spanner is ready" 08:14:27Z
08:14:39Z aborted / 08:14:50Z bus error / 08:15:02Z segfault / 08:15:14Z segfault
(intervals of 11.62 s, 11.61 s, 11.74 s), then quiet while idle
So the crashes occur whenever the process has work to do (its own initialisation is enough)
and are not caused by client traffic.
Crash evidence: this is memory corruption, and it is non-deterministic
The C++ binary writes its own crash dumps to /spanner/logs//STDOUT.*. Four crashes
from one idle container were collected. All four differ in fault address, stack, thread and
CPU:
instance crash time signal reported signal in fault address si_code pid/tid cpu
1 08:14:21Z 08:14:39Z aborted SIGSEGV 0x1fffffff0 1 (SEGV_MAPERR) 21/787 4
2 08:14:39Z 08:14:50Z bus error SIGBUS 0x61767265746e69 1 1086/1441 1
3 08:14:50Z 08:15:02Z segfault SIGSEGV 0x7330f302fa60 2 (SEGV_ACCERR) 1559/1974 7
4 08:15:02Z 08:15:14Z segfault SIGSEGV 0x132808080 1 (SEGV_MAPERR) 2037/2051 4
(Dump files are named after the instance's start time, so each row pairs one instance's dump
with the death the supervisor logged for it. In row 1 the supervisor reported "aborted" while
the dump records SIGSEGV; the crash handler re-raises the signal, so the exit status seen by
the parent does not always match the original one.)
Crash #2 is the most informative. The fault address decodes as ASCII text:
0x61767265746e69 -> little-endian bytes 69 6e 74 65 72 76 61 00 -> "interva\0"
and the reported PC is that same address (the stack trace begins with
"trace=61767265746e69,..."), i.e. control was transferred into string data. Crash #3 has the
same shape (PC == fault address). In the dump quoted below the loaded binary occupies
0xaaaadc200000-0xaaaafd40fbc0, so these fault addresses are far outside any mapping.
All stacks are ~24 frames deep and end in start_thread, i.e. the crashes happen on
background threads, not on request-handling threads. Symbols are stripped and the dumps
reference an internal Google symbol server, so we cannot resolve function names.
Example dump excerpt:
*** SIGSEGV (@0xe8075ff0e8075), si_code=1, see go/stacktraces#s15 received by
PID 252 (TID 337) on cpu 3; stack trace: ***
PC: @ 0xaaaafafa81ec (unknown) (unknown)
...
@ 0xffff8c67dd4c 288 start_thread
E0818 08:10:38 coredump_hook.cc:278] RAW: Remote crash gathering disabled
E0818 08:10:38 process_state.cc:704] RAW: Raising signal 11 with default behavior
A/B result: the kernel is the trigger
Everything except the Docker Desktop version was held constant (same image digest, same
compose.yaml, same host, same 25-minute test suite):
Metric kernel 7.0.12 kernel 6.12.76
spanner_server crashes 120 (24 min) 0 (25 min)
time to "Spanner is ready" 105 s 9 s
gRPC failures seen by clients 4 0
"connection reset by peer" to the DB 8 0
end-to-end test failures 4 0
The crash-free run under 6.12.76 was measured with two independent methods that agreed:
polling every 20 s (12 container generations tracked, since the suite recreates the
container between phases) and an event-driven collector that attached "docker logs -f" on
each "start" event so short-lived containers could not be missed.
What we ruled out
BUG:/WARNING:/Oops/call trace/corruption messages
Other containers on the same VM are unaffected
21 other containers were running on the same VM throughout, all with RestartCount=0 and no
OOM kills: three BigQuery emulators, Firestore emulator, fake-gcs-server, Envoy 1.32.1,
Jaeger, Cloud Spanner Emulator 1.5.55, an OpenTelemetry collector, and several Go services.
Only the heaviest process - a 660 MB C++ binary with ~1,500 mappings and many threads -
is affected. We mention this both as a data point and because it means we cannot show
VM-wide corruption.
Why this was hard to detect
The crash is invisible from Docker's point of view. The in-container supervisor restarts the
child within ~0.1 s, so:
docker inspect -> RestartCount=0, State.Status=running, Health=healthy
docker ps -> "Up 23 minutes (healthy)"
Nothing suggests a problem unless you read the container's stdout. We only found it by
tracing gRPC "DeadlineExceeded" errors in application logs back to
"error reading server preface: read tcp [::1]:58770->[::1]:15000: read: connection reset by
peer" - i.e. the database process disappearing mid-request.
Reproduction
us-docker.pkg.dev/spanner-omni/images/spanner-omni:2026.r1-beta.4
start-single-server
docker logs omni-repro | grep "has stopped"
Expected on 7.0.12 : several "signal: segmentation fault" / "bus error" / "aborted" lines
Expected on 6.12.76: no output
No ports, volumes or client traffic are required.
Impact
suspect Docker Desktop at first; time was spent investigating the workload's own
versions instead.
Questions and requests
why is it not in the release notes? Kernel changes are visible to workloads and should
be documented.
particular, is free page reporting (page_reporting.page_reporting_order=2, which is
passed on the kernel command line) known to be safe on this kernel in combination with
Apple Virtualization.framework?
around a kernel regression without downgrading the whole application?
versions, or a run with specific kernel parameters changed - we can reproduce on demand.
Current workaround
Staying on Docker Desktop 4.86.0 with automatic updates disabled
(DisableUpdate=true, AutoDownloadUpdates=false).
Attachments
Prepared as spanner_omni_crash_report.tar.gz (2.1 MB):
README.txt what is in the archive and how it was collected
01_crash_summary.txt the four crash signatures side by side, incl. the ASCII pointer
02_supervisor_log.txt the supervisor's record of the child deaths
03_environment.txt host / Docker / kernel / image digests / container layout
04_docker_version_diff.txt the measured 4.86.0 vs 4.87.0 comparison (kernel, desktop.img,
engine binary) and the update timeline from the install log
05_ab_result.txt the A/B measurement and how the 0-crash result was verified
crash_stacks/ the four STDOUT files containing a crash dump
dumps/ the complete, unmodified /spanner/logs tree from the container
Can also be provided on request:
(~/Library/Containers/com.docker.docker/Data/log/vm/console.log*)
Platform
macOS
Version information
Diagnostics ID
No response