Skip to content

feat(stages): handle partial trie unwinds - #26543

Merged
mediocregopher merged 11 commits into
mainfrom
mediocregopher/pipeline-partial-trie-unwind
Aug 4, 2026
Merged

mediocregopher merged 11 commits into
mainfrom
mediocregopher/pipeline-partial-trie-unwind

Conversation

@mediocregopher

@mediocregopher mediocregopher commented Jul 31, 2026 •

Copy link
Copy Markdown
Member

Extracts the staged pipeline unwind handling from #24100.

On startup, a lagging partial_state_trie frontier first unwinds to the durable trie tip with full changed-branch walks. A lower RocksDB/static-file target then runs as a separate normal unwind.

walk_all_changed_branch_children

This new flag is required when unwinding to partial_state_trie in order to handle the following scenario:

  • Block 10: hash of child 2 of trie node N is updated
  • Block 14: hash of child 4 of trie node N is updated
  • Block 10->12 is persisted, 13->14 masking
    • trie node N is not persisted, because it is masked
  • kill -9

In this situation we would unwind 14->12. However when doing the unwind only N's child 4 will be in the prefix set which is used; 2 will not be in the prefix set because it was only updated in block 10. Because of this a stale hash for 2 would get used, and any ancestor cached hashes would end up incorrect as well.

The fix is to enable this walk_all_changed_branch_children mode, under which the TrieWalker will visit all children of a branch when just one child of the branch is in the prefix set. This way any stale sibling hashes are also refreshed.

A durable metadata marker records the original Finish and partial-state-trie frontiers before recovery starts. Interrupted recovery resumes from this marker, which is deleted only after the special unwind completes successfully.

@mediocregopher mediocregopher added C-bug An unexpected or incorrect behavior A-staged-sync Related to staged sync (pipelines and stages) A-trie Related to Merkle Patricia Trie implementation labels Jul 31, 2026
@github-project-automation github-project-automation Bot moved this to Backlog in Reth Tracker Jul 31, 2026
@mediocregopher mediocregopher removed the C-bug An unexpected or incorrect behavior label Jul 31, 2026
@mediocregopher mediocregopher changed the title fix(stages): handle partial trie unwinds feat(stages): handle partial trie unwinds Jul 31, 2026
@mediocregopher
mediocregopher marked this pull request as ready for review August 3, 2026 09:08
@mediocregopher mediocregopher self-assigned this Aug 3, 2026
mediocregopher added a commit that referenced this pull request Aug 3, 2026
@mediocregopher

Copy link
Copy Markdown
Member Author

cyclops audit

@github-actions

github-actions Bot commented Aug 3, 2026 •

Copy link
Copy Markdown
Contributor

cc @mediocregopher

Cyclops audit event published. View workflow run

Config: config: default, iterations: default, hours: default

@tempoxyz-bot

Copy link
Copy Markdown
Member

👁️ Cyclops Security Review

eaa869d

🧭 Auditing · mode=normal · workers 0/2 done (2 left) · verify pending 0

Worker Engine Latest Status Status
pr-26543-w1 gpt-5.5 Not started Running
pr-26543-w2 claude-opus-5 Not started Running
⚙️ Controls
  • 🚀 Keep only 1 remaining iteration per worker after the current work finishes.
  • 👀 Keep only 2 remaining iterations per worker after the current work finishes.
  • ❤️ Let only worker 1 continue; other workers skip queued iterations.
  • 😄 Let only worker 2 continue; other workers skip queued iterations.
  • 🎉 End faster by skipping queued iterations and moving toward consolidation.
  • 😕 Stop active workers/verifiers now and start consolidation immediately.

@tempoxyz-bot tempoxyz-bot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👁️ Cyclops Review

Found 3 actionable findings, all in the new partial-trie startup recovery orchestration. The trie-walker flag propagation appears targeted, but the startup unwind scheduler needs stronger crash-consistency, target-readability, and pruning-retention checks before this recovery path is safe.

Reviewer Callouts
  • ⚡ Crash consistency of startup unwinds: The new recovery path relies on normal per-stage pipeline commits; review every crash point between FinishStage and MerkleUnwind for resumability.
  • ⚡ Partial persistence with pruning: Review how num_state_masking_blocks, persistence thresholds, and account/storage prune modes interact so crash-recovery history cannot be pruned away.
  • ⚡ Combined recovery ordering: Re-check mixed partial-trie and storage/static-file inconsistency cases, especially Headers, so the first queued target is always readable and lower repair targets are not skipped.
  • ⚡ Trie walker resumed stacks: with_walk_all_changed_branch_children does not recompute can_skip_current_node after constructing a walker from an existing stack; current partial recovery uses fresh walkers, but future resumed use of this mode should be audited.

Comment thread crates/node/builder/src/launch/common.rs Outdated
Comment thread crates/node/builder/src/launch/common.rs
Comment thread crates/node/builder/src/launch/common.rs Outdated
@github-project-automation github-project-automation Bot moved this from Backlog to In Progress in Reth Tracker Aug 4, 2026
@mediocregopher
mediocregopher added this pull request to the merge queue Aug 4, 2026
Merged via the queue into main with commit 0b0f7db Aug 4, 2026
34 checks passed
@mediocregopher
mediocregopher deleted the mediocregopher/pipeline-partial-trie-unwind branch August 4, 2026 12:58
@github-project-automation github-project-automation Bot moved this from In Progress to Done in Reth Tracker Aug 4, 2026
teyrebaz33 pushed a commit to teyrebaz33/tempo that referenced this pull request Aug 25, 2026
Automated nightly update of reth dependencies from `paradigmxyz/reth`
main branch.

## Upstream reth changes


[`10aa6a5...00ff650`](paradigmxyz/reth@10aa6a5...00ff650)

🔗 Amp thread:
https://ampcode.com/threads/T-01a036f1-9f58-7640-b789-4e3dd9778ebe
- **Engine**
- Improved payload building across canonical ancestors, finality,
persistence handoffs, pending resolution, and responsive cancellation
([#26559](paradigmxyz/reth#26559),
[#26567](paradigmxyz/reth#26567),
[#26580](paradigmxyz/reth#26580),
[#26708](paradigmxyz/reth#26708),
[#26759](paradigmxyz/reth#26759)).
- Added Bogota Engine API support and fork-time validation, plus FOCIL
inclusion-list construction and stubs
([#26682](paradigmxyz/reth#26682),
[#26706](paradigmxyz/reth#26706),
[#26711](paradigmxyz/reth#26711),
[#26737](paradigmxyz/reth#26737)).
- Tightened payload and block-access-list validation, including
state-gas admission and malformed BAL rejection
([#26651](paradigmxyz/reth#26651),
[#26694](paradigmxyz/reth#26694),
[#26719](paradigmxyz/reth#26719)).
- Fixed little-endian cell bitvectors, Osaka `getBlobsV4`, and
prewarm-worker shutdown
([#26650](paradigmxyz/reth#26650),
[#26703](paradigmxyz/reth#26703),
[#26768](paradigmxyz/reth#26768)).
- Reused scratch buffers for faster BAL hash encoding
([#26701](paradigmxyz/reth#26701)).

- **RPC**
- Added `debug_traceChain`, Alloy trace-chain result types, block-level
EVM reuse, and raw block transactions on the auth server
([#26582](paradigmxyz/reth#26582),
[#26614](paradigmxyz/reth#26614),
[#26669](paradigmxyz/reth#26669),
[#26760](paradigmxyz/reth#26760)).
- Added configurable response compression and request decompression
([#26668](paradigmxyz/reth#26668),
[#20277](paradigmxyz/reth#20277)).
- Added Bogota Engine API stubs and Amsterdam system contracts to
`eth_config` ([#26691](paradigmxyz/reth#26691),
[#26705](paradigmxyz/reth#26705)).
- Fixed cancellation of payload-hash and blocking-I/O work
([#26569](paradigmxyz/reth#26569),
[#26776](paradigmxyz/reth#26776)).
- Corrected execution-witness block identifiers, optional receipt
conversion/caching, and access-list environment preparation
([#26572](paradigmxyz/reth#26572),
[#26596](paradigmxyz/reth#26596),
[#26599](paradigmxyz/reth#26599),
[#26598](paradigmxyz/reth#26598)).
- Fixed testing block gas limits, transaction gas-limit preservation,
timestamp overflow, and chain-ID validation
([#26632](paradigmxyz/reth#26632),
[#26743](paradigmxyz/reth#26743),
[#26767](paradigmxyz/reth#26767),
[#26782](paradigmxyz/reth#26782)).
- Added network-specific log responses and generic testing RPC handlers
([#26491](paradigmxyz/reth#26491),
[#26547](paradigmxyz/reth#26547)).

- **Networking**
- Added ingress limits, configurable no-op client versions, and outbound
`GetCells` support
([#26660](paradigmxyz/reth#26660),
[#26652](paradigmxyz/reth#26652),
[#26673](paradigmxyz/reth#26673)).
- Improved handshake and protocol safety through ECIES identity checks,
message-ID validation, negotiated-protocol assertions, and bad-message
handling ([#26639](paradigmxyz/reth#26639),
[#26654](paradigmxyz/reth#26654),
[#26659](paradigmxyz/reth#26659),
[#26671](paradigmxyz/reth#26671)).
- Fixed ping/pong validation and pacing
([#26698](paradigmxyz/reth#26698),
[#26702](paradigmxyz/reth#26702)).
- Made eth/72 blob-cell announcements interoperable with geth while
preserving availability masks
([#26573](paradigmxyz/reth#26573),
[#26670](paradigmxyz/reth#26670)).
- Shared the snap/2 slim-account codec between client and server
([#26587](paradigmxyz/reth#26587)).

- **Txpool**
- Added blob-cell availability tracking and exposure on pooled
transactions ([#25463](paradigmxyz/reth#25463),
[#26642](paradigmxyz/reth#26642)).
- Included blob-pool transactions in queued counts and listings
([#26677](paradigmxyz/reth#26677),
[#26679](paradigmxyz/reth#26679)).
- Allowed senders with empty code hashes, added consensus encoding, and
converted sender accessors to iterators
([#26644](paradigmxyz/reth#26644),
[#26739](paradigmxyz/reth#26739),
[#26681](paradigmxyz/reth#26681)).

- **Trie & State**
- Added partial trie unwind and persistence support, including
changeset-cache handling
([#26543](paradigmxyz/reth#26543),
[#26612](paradigmxyz/reth#26612)).
- Corrected witness construction to use depth-first node order
([#26707](paradigmxyz/reth#26707)).
- Simplified `HashedPostState` wipe handling and added trie-data
reference collectors
([#26524](paradigmxyz/reth#26524),
[#26752](paradigmxyz/reth#26752)).

- **Storage & Providers**
- Removed `ConsistentDbView`, relocated `OverlayStateProvider`, and
simplified provider bounds
([#26581](paradigmxyz/reth#26581),
[#26611](paradigmxyz/reth#26611),
[#26591](paradigmxyz/reth#26591)).
- Fixed storage-wipe handling during batched persistence
([#26750](paradigmxyz/reth#26750)).
- Made RocksDB tolerate unknown column families
([#26647](paradigmxyz/reth#26647)).
- Updated static-file consistency checks to respect prune checkpoints
([#26565](paradigmxyz/reth#26565)).

- **Chainspec & Consensus**
- Added Bogota hardfork support and validated block-access-list hashes
during import ([#26686](paradigmxyz/reth#26686),
[#26696](paradigmxyz/reth#26696)).
- Honored the genesis `slotNumber` instead of hardcoding zero
([#26680](paradigmxyz/reth#26680)).
- Defaulted unspecified payload attributes
([#26684](paradigmxyz/reth#26684)).

- **DNS**
- Fixed EIP-1459 discovery records by rejoining long TXT character
strings and ignoring unrelated TXT records
([#26602](paradigmxyz/reth#26602),
[#26603](paradigmxyz/reth#26603)).

- **Snapshots & CLI**
- Added base-URL resolution and exposed prepared snapshot context
([#26576](paradigmxyz/reth#26576),
[#26777](paradigmxyz/reth#26777)).
- Fixed history downloads when the final snapshot chunk is partial
([#26607](paradigmxyz/reth#26607)).
- Added bootnode configuration to `reth.toml`
([#26551](paradigmxyz/reth#26551)).

- **Testing & Development**
- Improved engine reorg tests with explicit finality management and
expanded execute-blob Hive coverage
([#26584](paradigmxyz/reth#26584),
[#26606](paradigmxyz/reth#26606)).
- Added a persistent-datadir testing node and made dev-mined blocks
canonical immediately
([#26774](paradigmxyz/reth#26774),
[#26761](paradigmxyz/reth#26761)).

- **Bench**
- Restored metrics visibility in benchmark run configurations
([#26461](paradigmxyz/reth#26461)).

- **Dependencies & Releases**
- Updated Alloy to 2.4.x, `alloy-hardforks`/`alloy-eip7928` to 0.4.8,
and released Reth 2.5.0–2.5.1
([#26663](paradigmxyz/reth#26663),
[#26666](paradigmxyz/reth#26666),
[#26685](paradigmxyz/reth#26685),
[#26687](paradigmxyz/reth#26687),
[#26700](paradigmxyz/reth#26700),
[#26771](paradigmxyz/reth#26771)).

## Migrations

🔗 Amp thread:
https://ampcode.com/threads/T-01a036f2-23b2-77c0-8959-5c3f0f9df6c5
- Upgraded Reth to `00ff650`, Alloy to `2.4.1`, and related dependencies
to match their latest APIs.
- Renamed workspace lint keys from kebab-case to snake_case for updated
Cargo lint syntax.
- Removed the no-longer-needed crate recursion limit.
- Migrated hashed storage construction from the removed `from_iter` API
to direct struct initialization.
- Reused Reth’s prepared snapshot manifest, base URL, and data
directory, removing Tempo’s duplicate manifest discovery, fetching,
parsing, and path-resolution logic.
- Updated snapshot planning to handle Reth’s new `(plan, prepared)`
return value and execution’s optional prepared manifest.
- Updated `HashedPostStateProvider` implementations and callers for its
new fallible `ProviderResult` return type.
- Added the required RPC log associated type and identity `convert_log`
implementation for the updated receipt converter trait.
- Updated pooled transaction construction for the reordered transaction
field and new `blob_cell_availability` field.

[GitHub
Workflow](https://github.com/tempoxyz/tempo/actions/runs/32804636065)

---------

Co-authored-by: Alexey Shekhirin <github@shekhirin.com>
Co-authored-by: Alexey Shekhirin <5773434+shekhirin@users.noreply.github.com>
Co-authored-by: Matthias Seitz <19890894+mattsse@users.noreply.github.com>
Co-authored-by: Richard Janis Goldschmidt <701177+SuperFluffy@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-staged-sync Related to staged sync (pipelines and stages) A-trie Related to Merkle Patricia Trie implementation

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants