ci: run eest benchmark shards without a ramdisk

[taratorio]

Problem

The eest-spec-enginextests-benchmark-150m-{parallel,sequential} jobs intermittently kill their runner — the job log ends with make: *** Terminated followed by The runner has received a shutdown signal, the hosted-runner presentation of the VM exhausting memory. 5 of the last 54 benchmark-150m job instances died this way (e.g. this run on #22053, an unrelated branch the same morning, a sequential-variant instance), always at 95–100% of the test phase, independent of the PR under test.

Profiling the 150m shard (locally, pinned to CI parallelism) shows peak demand of ~16.8 GB against the runner's 16 GB: a 9.9 GB evm process footprint (live heap 4.9 GB, roughly doubled by default GOGC; dominated by opReturn return-data buffers and TemporalMemBatch write-set clones during the test_unchunkified_bytecode cases, which allocate 10.6–12.1 GB each) plus a 6.9 GB MDBX datadir sitting on the 8 GB ramdisk — tmpfs bytes and process bytes compete for the same RAM. 1075 of the shard's 1077 tests share one (fork, preAllocHash) group, so a single node's datadir grows for essentially the whole run and the peak lands at the end. Whether a run survives comes down to GC timing and randomized test order — hence the flakiness.

Prior investigation

#22325 (closed in favour of this PR) profiled the same OOM and pinned why the shard began flaking on Jul 7: the persistent code cache from #22154 grew the benchmark datadir by ~2 GB (5.65 → 7.40 GB peak), pushing the peak co-resident heap+datadir from ~12 GB to ~14+ GB — see the profiling comment. The FCU-less CodeStore growth is being capped separately in #22335; with the datadir off RAM, that growth no longer threatens the runner either way — the two changes are complementary.

Change

The ramdisk exists for shards that churn hundreds of short-lived datadirs, where create/unlink journaling dominates. The benchmark shards are the opposite shape (3 long-lived datadirs), so the ramdisk buys them no wall time — and costs them the RAM that OOMs the runner.

• tools/eest-spec-shards.yml: new per-shard key no-ramdisk: true, set on all 14 enginextests-benchmark-* shards. Opt-in-true on purpose: a false-valued key would be invisible to both jq's // default and GitHub expressions' loose == (null == false is true there).
• .github/workflows/test-eest-spec.yml: honors the key — skips creating the tmpfs. Nothing else sets ERIGON_EXECUTION_TESTS_TMPDIR/TMPDIR on Linux (the setup-erigon TMPDIR override is Windows-gated), so these shards exercise the env-var-unset path and their datadirs land in the runner's default temp dir (/tmp, on the same root SSD as $RUNNER_TEMP).
• tools/run-eest-spec-test.sh: the same key also skips the local (Darwin) auto-ramdisk for these shards — whose 2 GB default the 150m datadir (~7 GB) could not fit anyway. Local runs likewise fall through to the OS default temp dir.

Non-benchmark shards are unchanged.

Wall-time evidence (A/B on identical runners)

Dispatched run 29073818835 — the 150m shards with datadirs on SSD, vs the five most recent green ramdisk runs:

	ramdisk (5 runs)	no ramdisk	delta
150m-parallel, test phase	16m40s – 17m06s	17m09s	+1.8% vs mean
150m-sequential, test phase	11m14s – 13m06s	13m22s	+8% vs mean, within the baseline's own ±8% spread
150m-parallel, job total	20.3 – 22.5 min	20.0 min	inside range
150m-sequential, job total	17.0 – 17.7 min	17.3 min	inside range

Both A/B jobs passed, with ~7 GB more headroom at peak.

Benchmark semantics note: these are --time throughput shards, so datadirs-on-disk puts SSD writeback inside the measured path — per-test wall times (and any MGas/s derived from them) shift slightly at this PR's boundary; the A/B above bounds it at ~+1.8% for the 150m-parallel test phase. This is deliberate: production nodes run MDBX on SSD/NVMe with the OS page cache, so the post-PR numbers are more representative of real-world execution than tmpfs-backed ones. Treat pre-/post-PR timings as different baselines when comparing historical job logs.

Verification

• actionlint, shellcheck, bash -n clean; make lint clean.
• Matrix render (yq -o=json of the manifest) carries no-ramdisk through to matrix.*; row parsing verified for benchmark, stable, and race-regex shards.
• make eest-spec-enginextests-benchmark-1m-sequential run locally on Darwin through the new path: no auto-ramdisk created, datadirs in the default temp dir, all tests pass (1076/1076); a stable-shard control still creates the ramdisk.
• On this PR's CI: every benchmark shard's Create RAM disk step is skipped and its log prints the default-tmpdir routing, while stable shards keep tmpdir: /mnt/erigon-ramdisk; the first CI Gate run had all eest shards green including both 150m jobs.
• The no-ramdisk configuration was also validated end-to-end by the A/B run above before this PR.

No Go code changes; this is CI/tooling configuration, so the TDD cycle does not apply.

[Copilot]

Pull request overview

This PR reduces flakiness in EEST engine-x benchmark CI shards by avoiding tmpfs/ramdisk usage for shards whose datadirs are large and long-lived, preventing RAM pressure that can OOM GitHub-hosted runners.

Changes:

• Add a no-ramdisk per-shard flag to tools/eest-spec-shards.yml and enable it for all enginextests-benchmark-* shards.
• Update the EEST spec GitHub Actions workflow to disable the setup ramdisk for those shards and place datadirs under $RUNNER_TEMP on disk.
• Update tools/run-eest-spec-test.sh to honor no-ramdisk by skipping the Darwin auto-ramdisk for affected shards.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File	Description
tools/run-eest-spec-test.sh	Parses the new manifest flag and skips Darwin auto-ramdisk when no-ramdisk is enabled.
tools/eest-spec-shards.yml	Documents and introduces no-ramdisk, enabling it for benchmark shards.
.github/workflows/test-eest-spec.yml	Disables the setup ramdisk for no-ramdisk shards and sets ERIGON_EXECUTION_TESTS_TMPDIR to $RUNNER_TEMP.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[yperbasis]

Approving. A few non-blocking nits:

• PR body: these enginextests-benchmark-* shards are --time throughput shards, so moving them off tmpfs puts SSD writeback into the measured path — their headline MGas/s will shift (pass/fail unaffected). Worth a line so the change isn't later read as a regression.
• test-eest-spec.yml: the ramdisk: input uses !matrix.no-ramdisk while SHARD_NO_RAMDISK uses matrix.no-ramdisk == true. They agree for true/absent, but diverge if the key is ever written as a quoted string. !(matrix.no-ramdisk == true) on the input line makes both derive from one predicate.
• test-eest-spec.yml (optional): make eest-spec-${{ matrix.shard }} interpolates an expression into a run: block. Pre-existing and matrix.shard is trusted, but since you're already adding an env: block, SHARD: ${{ matrix.shard }} + make eest-spec-"$SHARD" would align with the recent zizmor template-injection hardening (#22363).
• Context (optional): a pointer to #22325's profiling comment + #22335 would preempt "why not also fix the code cache" — that's the FCU-less TblCodeCache growth, handled separately and not needed here once the datadir is off RAM.

[taratorio]

Thanks for the review — all four nits addressed:

• Benchmark semantics (--time on SSD): added a note to the PR body. Agreed it's worth stating explicitly — and framed the direction too: disk-backed datadirs put the same I/O in the measured path that production nodes have, so the post-PR numbers are more representative than the tmpfs ones, not a regression. The A/B bounds the shift at ~+1.8% for the 150m-parallel test phase.
• Predicate divergence (!matrix.no-ramdisk vs == true): gone as of f3e17ee — the run-step SHARD_NO_RAMDISK block was removed in the follow-up commit, so !matrix.no-ramdisk on the ramdisk input is now the only predicate in the workflow. On the quoted-string concern more broadly: bare booleans are the documented contract in the manifest header (a quoted "false" would also misbehave in jq's // default in run-eest-spec-test.sh, independent of the workflow). If you'd rather have that enforced than documented, I can add a type assertion for the key to check-eest-shard-coverage.sh — happy to do it here or as a follow-up.
• Template-injection hygiene: done in 27b2641 — SHARD env + make "eest-spec-$SHARD", matching the cache-warming step's TARGET pattern and the ci: fix zizmor template-injection in 5 QA RPC integration workflows (#21132) #22363 hardening.
• Context pointers: added a "Prior investigation" section to the PR body linking your profiling comment on cmd/evm, tools: bound enginextest tester datadir to fix eest 150m benchmark OOM #22325 (which pins the Jul-7 onset to the execution/cache, execution/commitment, db/state: consolidate cache stack (freelru + persistent code cache) #22154 code-cache datadir growth) and db/state/execctx: enforce CodeStore table cap on the commit write path #22335 as the complementary CodeStore cap.