Async persistence (explore only)#793
Draft
joostjager wants to merge 8 commits intolightningdevkit:mainfrom
Draft
Async persistence (explore only)#793joostjager wants to merge 8 commits intolightningdevkit:mainfrom
joostjager wants to merge 8 commits intolightningdevkit:mainfrom
Conversation
Switch from the sync MonitorUpdatingPersister to MonitorUpdatingPersisterAsync for ChainMonitor, enabling non-blocking channel monitor persistence at runtime. Previously, two separate persisters were created during build: an async one (used only to read monitors at startup, then discarded) and a sync one (passed to ChainMonitor::new for runtime persistence). Now a single AsyncPersister is used for both reading and ongoing persistence via ChainMonitor::new_async_beta. A DynStoreRef newtype is introduced to wrap Arc<DynStore> with a direct KVStore implementation, avoiding higher-ranked lifetime issues that arise when Arc<dyn DynStoreTrait> is used as a type parameter in complex generic bounds. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Change write_node_metrics from using KVStoreSync::write to KVStore::write().await. At all call sites where an RwLockWriteGuard was previously held across the persist call, split into two scopes: one for the in-memory mutation under the write lock, and a second that takes a read lock snapshot for the async persist. In the electrum chain source, inline the apply_wallet_update closure since closures cannot be async, restructuring the sync and async parts accordingly. In the builder, wrap the call in runtime.block_on() since it runs in a sync context during node construction. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Convert DataStore methods (insert, insert_or_update, remove, update, persist) from sync KVStoreSync to async KVStore, then propagate async through all callers: payment modules (bolt11, bolt12, spontaneous, unified), event handling, wallet, chain sync, and lib.rs. Apply Pattern B (do in-memory work under lock, drop lock, then await persist) throughout DataStore. Apply Pattern D in wallet (collect persistence operations under lock, drop lock, execute async) with block_in_place for the sync Listen trait impl. Remove the now-unused runtime field from Bolt11Payment since receive_via_jit_channel_inner can now directly .await instead of using runtime.block_on(). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace KVStoreSync with async KVStore in PeerStore, making persist_peers, add_peer, and remove_peer async. This propagates async through connect, disconnect, open_channel, open_announced_channel, close_channel, force_close_channel, and their internal helpers. In event.rs, restructure the ChannelPending handler to extract peer info under the network graph read lock, then drop it before awaiting add_peer (fixing a Send constraint issue). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace KVStoreSync with KVStore in static_invoice_store.rs, adding .await to the read and write calls in handle_static_invoice_requested and handle_persist_static_invoice respectively. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Convert the BDK wallet persistence layer from sync WalletPersister to async AsyncWalletPersister, continuing the migration away from KVStoreSync. Key changes: - Convert impl_read_write_change_set_type! macro and read_bdk_wallet_change_set in io/utils.rs from KVStoreSync to async KVStore - Implement AsyncWalletPersister for KVStoreWalletPersister in wallet/persist.rs - Change wallet persister field to tokio::sync::Mutex and add persist_wallet() async helper in wallet/mod.rs - Use load_wallet_async/create_wallet_async in builder.rs - Make splice_in, splice_out, new_address, send_to_address, and send_all_to_address async - Use block_in_place+block_on for sync trait impls (Listen, SignerProvider) - Add Async annotations in UDL bindings for newly-async methods Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove the SyncAndAsyncKVStore supertrait and all KVStoreSync implementations from SqliteStore, VssStore, and InMemoryStore. Simplify DynStoreTrait to only contain async methods, renaming read_async/write_async/remove_async/list_async back to read/write/remove/list. Remove the blocking_client field from VssStoreInner since it was only used by the KVStoreSync impl, and rename async_client to client. Remove sync test infrastructure (do_test_store, create_persister, create_chain_monitor) that depended on KVStoreSync. Convert remaining test helpers and integration test common module to use async KVStore via block_on. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
👋 Hi! I see this is a draft PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This changeset was generated with the help of AI tooling (Claude Code) to get an impression of the work involved in migrating to fully async persistence. It is not intended to be merged as-is.
This branch migrates all persistence operations in ldk-node from the synchronous
KVStoreSynctrait to LDK's asyncKVStoretrait, eliminating blocking I/O from the runtime entirely.The migration is broken into 8 incremental commits, each converting one subsystem and propagating async through its callers:
Use async ChainMonitor persister via
new_async_beta: Switch fromMonitorUpdatingPersistertoMonitorUpdatingPersisterAsync, using a single async persister for both startup reads and runtime persistence. IntroducesDynStoreRefto work around higher-ranked lifetime issues withArc<dyn DynStoreTrait>.Convert
write_node_metricsto asyncKVStore: Split in-memory mutations from async persist calls to avoid holding write locks across.awaitpoints.Convert
DataStorepersistence to asyncKVStore: The largest commit, propagating async through all payment modules (bolt11, bolt12, spontaneous, unified), event handling, wallet, and chain sync. Uses the pattern of doing in-memory work under a lock, dropping the lock, then awaiting the persist.Convert
PeerStorepersistence to asyncKVStore: Makes peer management operations async, restructuring event handlers to avoid holding non-Send locks across await points.Convert
StaticInvoiceStorepersistence to asyncKVStore: Straightforward async conversion of static invoice read/write operations.Convert BDK wallet persistence to
AsyncWalletPersister: Converts the BDK wallet layer from syncWalletPersistertoAsyncWalletPersister, switching the persister to atokio::sync::Mutex. Usesblock_in_place+block_onfor sync trait impls (Listen,SignerProvider).Convert
export_pathfinding_scoresto asyncKVStore.Remove
KVStoreSyncfrom core traits and store implementations: Final cleanup removing theSyncAndAsyncKVStoresupertrait, allKVStoreSyncimpls fromSqliteStore/VssStore/InMemoryStore, and the sync test infrastructure. SimplifiesDynStoreTraitto async-only methods.Across 31 files, 1301 insertions and 1516 deletions, the key recurring patterns are:
.awaitthe persistblock_in_place+block_onwhere sync trait boundaries require it (e.g.,Listen,SignerProvider)SqliteStorecalls inasync { ... .await }insideblock_onto ensure the tokio runtime context is available forspawn_blocking