Flaky: TestFromZero_BranchCacheCoherentAcrossBatches/parallel — GetAsOf on TemporalMemBatch with inMemHistoryReads disabled

[mh0lt]

Flaky test

TestFromZero_BranchCacheCoherentAcrossBatches/parallel — execution/execmodule/execmoduletester (added in #21952).

Run with the error: CI Gate run 28449207543, race-tests / tests-linux (ubuntu-latest, execution-other, serial). The job passed on retry within the same run (final job conclusion success), confirming it is intermittent.

Symptom

--- FAIL: TestFromZero_BranchCacheCoherentAcrossBatches (0.48s)
    --- FAIL: TestFromZero_BranchCacheCoherentAcrossBatches/parallel (0.24s)
        from0_genesis_internal_test.go:319:
            Received unexpected error:
            failed restore state : GetAsOf called on TemporalMemBatch with inMemHistoryReads disabled

Analysis

The error itself is a deterministic mode guard (TemporalMemBatch.GetAsOf returns it when inMemHistoryReads is false), so this is not value corruption — it's a timing race that surfaces as the guard when it loses:

• The test re-executes from 0 in 1 KB batches via the integration path, which sets SetInMemHistoryReads(false) on the SD (cmd/integration/commands/stages.go).
• Each batch calls SeekCommitment at the start of ExecV3 (exec3.go:122), with inMemHistoryReads still false. The restore path is SeekCommitment → restorePatriciaState → hext.SetState(...), which can issue a GetAsOf.
• In parallel mode the previous batch's commit is asynchronous (commit goroutine + commitResults). If the prior batch's mem-batch flush hasn't completed when the next batch's SeekCommitment runs, the GetAsOf hits the still-populated TemporalMemBatch while inMemHistoryReads is false → the guard fires.
• Parallel exec only enables inMemHistoryReads inside the executor (exec3_parallel.go:240, capturing/restoring the caller's value — added for parallel commitment calculations implemented #20805 against this same error string), which is after the start-of-batch SeekCommitment.

So the trigger is an ordering race between batch N's async commit/flush and batch N+1's pre-exec SeekCommitment restore. Timing-dependent → intermittent.

Local repro attempts (all green)

On mh/perf-statecache-lru-pr HEAD, execution/execmodule/execmoduletester:

• -race -run TestFromZero_BranchCacheCoherentAcrossBatches: 6/6 pass
• -race -count=12 (single test): pass
• -race -count=10 at GOMAXPROCS=1, 2, 4: pass
• full package -race, 4×: pass

Could not reproduce locally; it appears to need the concurrent-package CPU pressure of the race-tests group to widen the window.

Notes

• Pre-existing: both the test (cmd/integration: stage_exec to not reuse doms object #21952) and the racing code (exec3_parallel.go, exec3.go) are on main; not introduced by a cache change. Surfaced during execution/cache: StateCache LRU + Mode + (txNum,epoch) lazy unwind #21386's CI.
• Not skipping per repo policy — filing to track. The likely fix is to ensure the prior batch's commit/flush completes (or inMemHistoryReads is consistent across the SeekCommitment-then-parallel-exec boundary) before the next batch's restore.

cc parallel-exec / commitment owners.

[mh0lt]

Second occurrence, on a different PR (#21536, vio-on-main): CI Gate run 28474381768, race-tests / tests-linux (ubuntu-latest, execution-other, **parallel**) — same assertion failed restore state : GetAsOf called on TemporalMemBatch with inMemHistoryReads disabled in TestFromZero_BranchCacheCoherentAcrossBatches/parallel. Confirms it's independent of either PR's changes (it hit both #21386's -serial race shard and #21536's -parallel one) and surfaces under the race group's concurrent-package load. Both recovered on retry. Raising priority — it's now intermittently red-flagging unrelated PRs.

[VBulikov]

yperbasis added Imp1 as Importance