Skip to content

fix(database): abort P39 cascade recovery when pop_block() makes no progress (last write-lock deadlock loop) - #118

Merged
On1x merged 1 commit into
masterfrom
fix/p39-cascade-pop-no-progress-deadlock
Jun 16, 2026
Merged

fix(database): abort P39 cascade recovery when pop_block() makes no progress (last write-lock deadlock loop)#118
On1x merged 1 commit into
masterfrom
fix/p39-cascade-pop-no-progress-deadlock

Conversation

@chiliec

@chiliec chiliec commented Jun 16, 2026

Copy link
Copy Markdown
Member

Summary

Follow-up to #117. That PR fixed the write-lock deadlock in the three fork-switch pop loops of _push_block(), but a fourth pop loop — the P39 cascade-recovery path (libraries/chain/database.cpp, the while (head_block_num() > original_head && head_block_num() > lib_num) loop) — was left unguarded. It carries the identical deadlock.

This PR closes it, so no unguarded no-progress pop loop remains in _push_block().

The bug

When pop_block()'s undo() is a no-op — a committed/empty undo session while head is still above LIB — head_block_num() never decreases. Neither loop condition (> original_head, > lib_num) can ever become false, so the loop spins forever holding the global chainbase write lock (writer_held_ms climbing for hours), starving every reader and the validator's own block-production loop. This is the same wedge observed in the minority-fork reorg incident that motivated #117.

The fix

Mirror #117's recovery-loop pattern exactly: capture head_block_id() before pop_block(), and if it is unchanged afterward, break. The _fork_db.reset() + start_block() that already run immediately after this loop restore fork_db to the database head, so breaking leaves consistent state. Minimal, no behavior change on the healthy path.

Verification status

⚠️ Not yet built or run — opened as a draft. Needs a local build + a soak on a validator/RPC node before merge. The change is a 1-loop mirror of the pattern already merged and proven in #117, but consensus-path code warrants a real build/test pass.

Known follow-ups (out of scope here — deliberately not patched blind)

These do not cause the deadlock and are noted for tracking rather than fixed speculatively:

  1. pop_block() advances fork_db before the no-op undo(). pop_block() runs _fork_db.pop_block() then undo(). On a no-op pop the fork_db head moves back one while the chainbase head does not, so the abort paths in fix(database): abort fork switch when pop_block() makes no progress (write-lock deadlock) #117's main-switch branch (_fork_db.remove(new_head) + throw) can leave fork_db and chainbase desynced by one block. Likely self-heals on the next resync; worth confirming. (This loop is unaffected — it resets fork_db right after.)
  2. Abort ≠ convergence. Aborting a reorg leaves the node on its current (possibly minority) fork. In the snapshot-floor case where undo history is genuinely unavailable for the divergence range, the node stays alive and responsive but may not converge automatically and can still need a restart (which reopens at LIB and undo_all()s cleanly). The fix prevents the catastrophic lock-starvation; it does not guarantee automatic fork convergence.
  3. Regression test. A test that drives a fork switch / cascade recovery with an empty undo stack above LIB and asserts the write lock is released would lock in all four guards. Not added here pending familiarity with the test harness.

Possible cleanup

The no-progress guard now appears in four near-identical copies. A small private helper (e.g. a guarded pop returning false on no-progress) would de-duplicate them — left out here to avoid churning the freshly-merged #117 loops.

…rogress

#117 added a no-progress guard to the three fork-switch pop loops in
_push_block() but left the fourth pop loop — the P39 cascade-recovery
path — unguarded. It has the identical deadlock: when pop_block()'s
undo() is a no-op (committed/empty undo session while head is still
above LIB), head_block_num() never decreases, so neither
'> original_head' nor '> lib_num' can ever become false and the loop
spins forever holding the global chainbase write lock — the same wedge
(writer_held_ms climbing for hours) observed during the minority-fork
reorg incident.

Mirror #117's recovery-loop pattern: detect that head_block_id() did
not change after pop_block() and break. The existing _fork_db.reset()
+ start_block() right after the loop already restores fork_db to the
database head, so break leaves consistent state. This closes the last
unguarded no-progress pop loop in _push_block().
@chiliec
chiliec marked this pull request as ready for review June 16, 2026 04:47

@On1x On1x left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

@On1x
On1x merged commit 5d2861d into master Jun 16, 2026
1 check passed
@chiliec
chiliec deleted the fix/p39-cascade-pop-no-progress-deadlock branch June 16, 2026 05:26
web3blind pushed a commit to web3blind/viz-cpp-node that referenced this pull request Jul 3, 2026
Observability backstop for the global chainbase write lock. A wedged push_block()
holding the write lock takes the whole node down (every reader + the validator's
own block production block on it; observed: writer_held_ms climbing for hours
during a minority-fork reorg).

A monitor thread watches the per-push_block hold time and, when a single
push_block holds the write lock past PUSH_BLOCK_STALL_WARN_SEC (60s) with no
operation applied in that window, emits a loud, throttled stderr diagnostic. It
also exposes push_block_lock_held_ms() for external healthchecks.

LOG-ONLY by design: it never calls std::_Exit and never mutates state, so a false
positive can only add a log line, never a node death. A force-exit watchdog was
deliberately rejected: on a deterministic heavy block (e.g. a future hardfork) it
would self-destruct the whole network at once -- far worse than the localized hang
it would guard against. Restart decisions stay with the operator / orchestrator,
driven by push_block_lock_held_ms().

Complements the pop-loop no-progress guards (VIZ-Blockchain#117/VIZ-Blockchain#118): those fix the known
infinite loops; this only makes any remaining/future write-lock stall visible.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants