[qu0b]

Follows #1486 and #1487. The shadowfork snapshot downloader now fetches the archive as parallel ranged GETs, emitted in order into the same tar -I zstd -x. Nothing changes about snapshot format or layout.

Why

One HTTP stream from R2 through Cloudflare tops out around 95 MB/s. The link isn't the limit. Measured on a mainnet snapshot, from two different hosts:

workers	throughput
1	50–95 MB/s
8	302 MB/s
16	382 MB/s

At one stream, a 1.16 TB geth mainnet snapshot is a ~3.5 h download. At 16 workers it's under an hour.

How

• 16 workers each fetch 64 MiB ranges. An emitter writes the chunks to stdout in order.
• At most 2 × workers chunks (2 GiB) are staged in /tmp, so the extracted datadir is still the only thing on the volume.
• Resilience:
  • Partial chunks resume instead of restarting. Each request starts at the chunk's absolute archive offset plus the number of validated bytes already saved. Only a matching 206 Content-Range response is retained.
  • Fast straggler recovery adapts to slower links. Each worker starts with --speed-limit 1048576 --speed-time 30. If a timeout made valid progress, the worker preserves those bytes and halves its threshold, retaining that threshold across subsequent chunks. The floor is --speed-limit 1024 --speed-time 120. On a 100 Mbit/s uplink shared by 16 workers, the first timeout preserves the partial chunk and reduces the threshold to 512 KiB/s, allowing the remaining bytes to complete on the slower connection.
  • Progress does not exhaust the retry budget. SNAPSHOT_DOWNLOAD_MAX_ATTEMPTS limits consecutive attempts without validated progress; saved progress resets the counter.
  • Invalid Range responses cannot corrupt saved progress. A 200, mismatched Content-Range, or oversized response is discarded by restoring the partial file to its previous validated length. --max-filesize bounds each response, including bodies without Content-Length with the curl version supplied by Alpine 3.19.
  • Download or extraction failures fail the step loudly instead of marking an incomplete datadir finished.
• network_params.shadowfork_download_timeout controls the overall restore wait and defaults to 72h, allowing slower links and participants sharing one uplink more time.

Two POSIX-sh traps are handled and commented in the script:

• The emitter's progress counter is written with a rename. A torn read is an arithmetic error, and that silently exits sh; this is what hung an earlier version.
• The EXIT trap kills only the workers. kill 0 would also take down the tar the stream is piped into.

Verification

• Earlier verification of the parallel downloader rendered the script from the .star file and ran it in alpine:3.19.1, with 4 MiB chunks so there were many of them. It downloaded and extracted a 448 MB snapshot, and the extracted tree's hash matches the source.
• The same logic as a standalone script, also under alpine:3.19.1 busybox:
  • a 4 GiB prefix of a 374 GB snapshot is byte-identical to a single curl of the range;
  • per-GiB SHA-256s match the ones recorded while the object was uploaded.
• kurtosis lint passes.

I haven't run a full mainnet shadowfork enclave with this change yet.

https://claude.ai/code/session_01LMoaDSkP7fGBZGm25FZc5x

Supersedes #1488: that PR fixed failure modes of the single-stream resume loop this one replaces; see the mapping in its closing comment.

[redpandabot]

Summary

Replaces the single resumable curl stream with 16 workers fetching 64 MiB ranges, emitted in order into the same tar -I zstd -x. I simulated the script under dash with stubbed ranged curl: output is byte-identical and in order, the 2×workers staging bound holds, the flow-control gate cannot deadlock, and a failed chunk makes tar fail on truncation so the step exits loudly (no /tmp/finished). I also confirmed the cold-Range 200 hazard exists on snapshots.ethpandaops.io today and that --max-filesize guards it. One latent regression: the chunk GETs dropped the -L the old loop had, so a redirecting (user-configurable) snapshot base would fail after ~42 min of retries.

Issues

• 🟡 src/network_launcher/shadowfork.star:34 — chunk GETs dropped -L, so a redirecting snapshot base retries forever — see the thread on that line

---

_{Reviewed @ e2554623
"Don't drink the bear's skin before you've shot it." — Hungarian}

[redpandabot]

🟡 chunk GETs dropped -L, so a redirecting snapshot base retries forever

The old resume loop used curl -sfL, and the TOTAL probe still uses -sfIL, but the ranged chunk fetches use plain curl -sf: a 3xx is not an error for -f, so every chunk would fetch a redirect body, mismatch the size check, and retry 500×5 s before failing. The default endpoint returns 200 directly (verified), but network_sync_base_url is user-configurable, so re-adding -L (and ideally --proto-redir or -w '%{http_code}' to fail on 3xx) keeps the old behaviour.

[qu0b]

Fixed in b344b48: the chunk fetches are curl -sfL again. The default host is affected too when written as http://: it answers 301 → https. Measured on that redirect:

• without -L: exit 0 and a 167-byte redirect body;
• with -L: 206 and exactly the requested bytes, since curl re-sends the Range.

End to end in alpine:3.19.1, a 256 MiB prefix fetched through the http:// URL with 8 workers and 32 MiB chunks has the same sha256 as a direct https range fetch. No --proto-redir: a redirect off-scheme still has to deliver exactly the requested length, or the chunk is retried and the step fails loudly.

[redpandabot]

Addressed. Commit b344b48 re-adds -L to the ranged chunk fetch — src/network_launcher/shadowfork.star:34 now reads curl -sfL --connect-timeout 20 --speed-limit 1024 --speed-time 120 --max-filesize "$want" -r "$s-$e" ..., matching the old resume loop's -sfL behaviour, and curl re-sends the Range on the redirected request, so the size check still guards the delivered bytes.

Two notes on how the fix holds up, both grounded in the code as it is now:

• My original comment also suggested optionally failing fast on a 3xx with --proto-redir or a -w '%{http_code}' check. That was deliberately not taken: the final defence is the exact-length check at src/network_launcher/shadowfork.star:38 — a redirect to another scheme or off-host still has to hand back exactly want bytes or the chunk retries and eventually touch "$T/failed" fails the run, so the "retries forever with a redirect body" failure mode is closed regardless of where the redirect points. Reasonable trade-off; I'm fine with it.
• The redirected-response path also interacts with --max-filesize "$want" in a way worth noting: for a 3xx answered without following (i.e. no -L), curl 8.x applies --max-filesize to the response it is about to write, so a small redirect body still passes and would have been size-checked as before — which is exactly the retry-500-times scenario from the original comment. With -sfL restored that path is no longer reachable for the chunk fetches, so it's moot here; mentioning it only because the TOTAL probe at line 24 still relies on -I + header parsing and remains correct for the same reason.

---

_{@ b344b481}

[redpandabot]

The PR replaces the single resumable curl stream with 16 parallel ranged-GET workers feeding an in-order emitter into the same tar -I zstd -x pipeline. I re-traced the final script and empirically simulated the worker/emitter protocol in POSIX sh: ordered emission holds under heavy backpressure (N=100, window 2P=8), the failed flag propagates loudly through stream|tar, P>N is safe, and the size-only chunk check cannot be fooled by stale or poisoned .part files. The previously open finding — chunk GETs dropped -L, so a redirecting snapshot base would retry forever — is fixed by commit b344b48 (curl -sfL on the chunk fetch).

---

_{Reviewed 1 changed file(s) @ b344b481 — no blocking issues found.
"Don't drink the bear's skin before you've shot it." — Hungarian}

[qu0b]

@barnabasbusa could you take a look when you have a moment?

Review of b344b481:

Checked, looks correct

• Ordering and flow control: worker k handles chunks k, k+P, … in sequence, so the chunk the emitter is waiting for is always the one its worker is fetching or just finished. The i - emitted < 2P gate can't deadlock, and at most 2P × 64 MiB (2 GiB) is staged.
• Failure path: a failed worker sets $T/failed. The emitter returns 1, tar/zstd hit a truncated frame, and set -e fails the step. /tmp/finished is never written, so the wait doesn't pass.
• Range edges: the last chunk is clamped to TOTAL-1. --max-filesize only rejects bodies larger than the limit, so a 206 of exactly want bytes passes, while a 200 with the full body is rejected. -L is back on the chunk fetch.
• The failing Check consensus spec values jobs aren't related to this PR. Upstream added EIP8198_FORK_EPOCH/_VERSION, MIN_BLOB_DATA_RETENTION_MS and SLOT_DURATION_SCHEDULE, which feat: add EIP-8198 quick slots params #1491 covers.

Worth considering (non-blocking)

1. Head-of-line blocking from one slow connection. --speed-limit 1024 --speed-time 120 only catches near-zero progress. Because output is emitted in order, a single chunk crawling at 10 KB/s passes that check, but 64 MiB at that rate takes about 1.9 h. Meanwhile the 2 GiB window fills in about 6 s at 382 MB/s and the other 15 workers sit idle. With one stream, a slow connection only slowed things down. Here it stalls everything. A retry now costs just one chunk, so a much tighter limit is cheap, e.g. --speed-limit 1048576 --speed-time 30 or a per-chunk --max-time 300.
2. 2 GiB of staging in the container's /tmp, i.e. node ephemeral storage, per participant. The old version used none. On k8s with several shadowfork participants on one node, that's N × 2 GiB of unrequested ephemeral storage and a possible DiskPressure eviction. Fine for docker, but it may be worth a comment or setting P/C lower for k8s.
3. Nit: after the pipeline, [ ! -f "$T/failed" ] || exit 1 would make failure independent of zstd catching the truncation. That only matters if a multi-frame archive happened to end on a 64 MiB boundary, so it's very unlikely, but it costs one line.

As the description says, this still needs a real mainnet (or hoodi/sepolia) shadowfork enclave run before merge.

[redpandabot]

Summary

The shadowfork downloader now fetches the snapshot as 16 parallel ranged GETs (64 MiB chunks, resumed by absolute offset, emitted in order into the same tar stream) with a configurable plan.wait timeout. I verified all three previously-open findings are fixed: chunk GETs carry -L again; tar failure now fails the step (stream | tar … & + wait $! under set -eu, errexit on a failed wait confirmed in dash, with the frame-boundary case covered by the $T/failed check); and curl 8.5 (Alpine 3.19) enforces --max-filesize on every body write independent of Content-Length (lib/sendf.c cw_download_write), with the Content-Range equality check as a second gate. I rendered the script and ran it against a local Range server: 206 success, dropped-connection resume, backpressure with 12 chunks/window 4, and persistent-200 refusal (loud failure, no /tmp/finished) all behave correctly with byte-identical output. One remaining defect: the EXIT trap kills the wrapper subshells but not the inner worker subshells.

Issues

• 🟡 src/network_launcher/shadowfork.star:30 — EXIT trap kills the wrapper subshells, not the worker subshells — see the thread on that line

---

_{Reviewed @ 9b05da6c
"Don't drink the bear's skin before you've shot it." — Hungarian}

[redpandabot]

Parallel ranged-GET snapshot downloader with in-order emission into tar, plus a configurable download timeout. All four previously open findings are fixed in the current head: -L is back on chunk GETs; tar/zstd failure now aborts via wait $! under set -e (verified empirically, /tmp/finished is not touched); non-206/mismatched responses are truncated back to validated length so a 200 without Content-Length cannot corrupt progress; the worker subshell is the $! the EXIT trap kills and its own trap kills the active curl. I ran the rendered script end-to-end against a local ranged-GET server (12 chunks, 4 workers, with mid-transfer connection drops): the extracted tree matched the source and /tmp/finished was only touched on success. No new issues found.

---

_{Reviewed 4 changed file(s) @ 9a986b36 — no blocking issues found.
"Don't drink the bear's skin before you've shot it." — Hungarian}