Repository navigation
test(godis): retry a crashed emu run before failing the differential gate - #372
Merged
Merged
Conversation
…gate The GoDis locked differential gate (TestE2ELocked) has been intermittently failing unrelated PRs with a LOCKED REGRESSION. The cause is not a godis codegen bug: the emulator faults nondeterministically on a small, run-varying set of corpus programs — under BOTH -c0 and -c1 (e.g. one run c1 segfaults on buffered_chan, another c0 segfaults on nested_fn), always ~1 of 266, at addresses in the 0x1000000xx family. That is a heap-layout-dependent memory corruption in the emu runtime shared by both modes, latent (master was green on the identical emu binary hours earlier), not something a given PR introduced. A single stray fault makes one mode print a crash marker while the other returns correct output, so Classify() sees c0!=c1 (JITMismatch) and the locked gate fails the build. Retry a crashed emu run up to maxEmuAttempts (3) times. A fresh process almost never reproduces the nondeterministic fault, so the flake clears; a genuine deterministic fault crashes on every attempt and is still reported — the gate keeps its teeth. runEmu's single-run body is split out as runEmuOnce. This unblocks CI while the underlying emu runtime corruption is fixed separately (x86-64 core-dump investigation; the fault is in the allocator/GC/loader, not the AMD64 JIT as first suspected — the interpreter faults too). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The GoDis (Go → Dis) Build & Test job's locked differential gate (
TestE2ELocked) has been intermittently failing unrelated PRs (#361, #370, and any fresh PR) with aLOCKED REGRESSION. It's a required check, so it randomly blocks merges and forces admin-bypass toil.Root cause (corrected)
It is not a godis codegen bug and not the AMD64 JIT stack-alignment issue first suspected. The emulator faults nondeterministically on a small, run-varying set of corpus programs under both execution modes:
buffered_chan.go-c1(JIT)0x100000010math_powmod.go-c1(JIT)0x7ffc…nested_fn.go-c0(interpreter)0x100000008The interpreter faulting rules out a JIT-only cause — this is heap-layout-dependent memory corruption in the emu runtime shared by both modes. It's latent (master was green on the identical emu binary hours earlier; a run just has to get unlucky), so no individual PR introduced it.
A single stray fault makes one mode emit a crash marker while the other returns correct output →
Classify()seesc0 != c1→JITMismatch→ the locked gate fails.Fix
Retry a crashed emu run up to
maxEmuAttempts(3). A fresh process almost never reproduces the nondeterministic fault, so the flake clears — while a deterministic fault (a genuine regression) crashes on every attempt and is still reported. The gate keeps its teeth; it just stops failing on noise.runEmu's single-run body is split out unchanged asrunEmuOnce.This PR's own GoDis run exercises the retry, so it self-validates.
Not fixed here
The underlying emu runtime corruption still needs a proper fix — an x86-64 core-dump investigation (the minipc), now aimed at the allocator/GC/Dis-loader rather than
comp-amd64.c. This PR only stops it from blocking every unrelated PR.🤖 Generated with Claude Code