Repository navigation
IBD: recover or abort on io_uring drain stall (#421) - #425
Merged
Merged
Conversation
Inner strings like "invariant: io_uring undrained" lack "corrupt record", so from_err_str treated them as Cascade. Recover/halt need EngineFault. Pin each session-fault string on StoreError and the confirm reject map. Co-authored-by: Cursor <cursoragent@cursor.com>
A 5s spin used to return with leftover SQEs so callers dropped in-flight buffers. Every UringSession now uses DrainBudget: progress resets the zero-CQE clock, slow drains log, and zero completions for RBITCOIN_URING_DRAIN_HARD_SECS (default 120) abort. Tests hit HardCap as undrained instead of abort(). Co-authored-by: Cursor <cursoragent@cursor.com>
Class C leftover above tip is the same repair as open. Credit is a tip gap so a recovered batch of 144 does not mint a new recover. Exhausted does not repair. Drops only this thread's TLS ring. Co-authored-by: Cursor <cursoragent@cursor.com>
Write requeues the same BQ heights after a credited recover; lookup does the same instead of warning every 20s. Credit exhausted or a drain stall aborts and names RBITCOIN_IO=pread. Failed head drain re-notes the taken pending list. ibd: perf gains uring_recover, slow_drain, and lookup_faults. Co-authored-by: Cursor <cursoragent@cursor.com>
Operators saw 19h of undrained warnings under idle IO. Document the 120s hard-cap abort, 1000-height recover, Class C repair, and that there is no mid-run libc fallback. Co-authored-by: Cursor <cursoragent@cursor.com>
Darwin SessionBackend is Pool-only, so `_` in drain_all_pool was unreachable under -D warnings and failed macos CI. Co-authored-by: Cursor <cursoragent@cursor.com>
Assumed 100ms windows let EINTR spin hit the hard cap. note() now takes a real dt. submit_and_wait_one shares that budget. Session Drop waits then poisons without abort(). Session-fault match is any Corrupt io_uring message except Linux-only/unavailable. drop_thread_local is gone: poisoned TLS already reopens. Co-authored-by: Cursor <cursoragent@cursor.com>
Lookup must not clear strong bits while write sets them before the fence. In-process recover no longer runs Class C repair. Two threads cannot both pass the tip-gap check. Co-authored-by: Cursor <cursoragent@cursor.com>
ConsensusError::is_uring_session_fault covers leftover cqe and submit failed. Head insert returns the taken batch on Err so write does not clone the drain list on the success path. Co-authored-by: Cursor <cursoragent@cursor.com>
One helper returns Option recover. Write classify takes that Option. Lookup EngineFault skips emit when ready has no hash. Load stamp/pin and scripts requeue on credited recover. Tip connect retries instead of note_invalid_block. Co-authored-by: Cursor <cursoragent@cursor.com>
Open still repairs leftover strong. In-process recover does not. Lookup 8 consecutive Io dequeues BQ; session-fault recover does not. Co-authored-by: Cursor <cursoragent@cursor.com>
rearden-grok
Bot
force-pushed
the
ibd/uring-recover
branch
from
September 11, 2026 19:44
ceba722 to
56e8f19
Compare
WaitOneWindow::Enter is only built from io_uring enter errors. IOCP wait_one_cqe had no callers after the timeout drain loop. Both failed -D warnings on macos/windows CI. Co-authored-by: Cursor <cursoragent@cursor.com>
Write, load, scripts, lookup, and tip-connect each matched Recovered vs Exhausted. Credit/abort lives on Query; IBD requeue sites share one helper. Lookup's 8-strike halt no longer carries recover arms. Co-authored-by: Cursor <cursoragent@cursor.com>
This was referenced Sep 11, 2026
Closed
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #421: a 5s
drain_allused to return with leftover SQEs, callers dropped in-flight buffers, and lookup warnedundrainedevery ~20s for hours. Completions were late/starved, not on-disk corruption.store: io_uring drain slowafter 5s; 120s of zero CQE progress aborts (buffers still pinned). Env:RBITCOIN_URING_DRAIN_HARD_SECS.pread. Abort namesRBITCOIN_IO=preadfor the next start. Session-fault strings classify asEngineFault, not Cascade.Test plan
cargo test -p rbitcoin-store uring(drain budget, leftover pending, unmatched CQE, session-fault strings)cargo test -p rbitcoin-query uring_recovercargo test -p rbitcoin-net --lib ibd::confirm::tests(write recover/abort, lookup policy, perf tokens)fmt,deny,clippy,ast-grep,test,windows,macos,multinode,coverage)Made with Cursor