confirm: walk past taken BQ hole; loadq=14 writeq=14

CHANGELOG.md

@@ -18,8 +18,18 @@ before 1.0).
   `SHA256SUMS` (90 days). Not required checks. `musl` uses the same
   label. Linux musl stays Nix; Darwin/Windows are native runners.
 
+### Changed
+
+- **Confirm queue caps:** `loadq=14` · `scriptq=4` · `writeq=14`
+  (was loadq=8 · writeq=20).
+
 ### Fixed
 
+- **Lookup walk after loadq take:** `block_queue_unresolved_heights`
+  starts after `lookup_taken_hi`. Taken BQ rows are not a fetch hole,
+  so lookup can fill loadq ahead of tip. A missing height *above*
+  that high-water still stops the walk.
+
 - **Windows / Darwin store smoke `--release` compile:** `take_raw_clone_n`
   and the raw-clone meter are `cfg(any(test, debug_assertions))`. Native
   `cargo test --release -p rbitcoin-store --lib` (windows.yml / macos.yml)

OPERATOR.md

@@ -233,7 +233,7 @@ pages after annotate does not protect `txout`. See
 | Bulk store IO | **uring** (Linux) when available | `RBITCOIN_IO` only; ring depth **128**. Segmented `tx.head` FdOnly; Class C L2 write-behind (`docs/io-modality.md`) |
 | Archive Class A append | **pwrite** (always) | `txout` / `inwit` / `spent` + `*.idx` mega-appends use `write_at_pwrite` only |
 | `tx.head` (segmented) | fixed geometry | Default **25-bit** heads (128 MiB) with **4 B relative** fks; roll at 80% load / body soft span; **binary fuse8** on seal. Legacy mono-head datadirs require reindex |
-| Confirm stages | **lookup · load · scripts · write** | Real queues **loadq=8 · scriptq=4 · writeq=20**. Lookup takes BQ in height order (stops at a hole), dequeues raw, and fills loadq with load-sized batches (8000 inputs / 144 blocks). IBD **lookup** TipOnly-resolves at most **64000** inputs or **1080** BQ-ready heights; hard **min 8000** when more unresolved heights remain. Wave table: [`docs/concurrency.md`](docs/concurrency.md). |
+| Confirm stages | **lookup · load · scripts · write** | Real queues **loadq=14 · scriptq=4 · writeq=14**. Lookup takes BQ in height order (stops at a hole after `lookup_taken_hi`), dequeues raw, and fills loadq with load-sized batches (8000 inputs / 144 blocks). IBD **lookup** TipOnly-resolves at most **64000** inputs or **1080** BQ-ready heights; hard **min 8000** when more unresolved heights remain. Wave table: [`docs/concurrency.md`](docs/concurrency.md). |
 | Confirm batch inputs | **8000** soft | Hardcoded. Live line: `h= n= in=` (**n** = blocks in pack, **in** = Σ inputs) |
 | Mempool weight budget | **~300e6 WU** | `--mempool-size-mb N` (maps N×1e6 WU) |
 | Inhibit auto-suspend | **off** | `--inhibit-suspend` (uses `systemd-inhibit` if available) |

crates/rbitcoin-net/src/ibd/confirm/mod.rs

@@ -372,13 +372,13 @@ pub(crate) fn pack_stop_after(
     n_blocks >= hard_max_blocks || sum_inputs > soft_max_inputs
 }
 
-/// Default lookup→load depth (`loadq`): one lookup-wave of prefetch (8×8000).
-pub(crate) const LOAD_QUEUE_CAP_DEFAULT: usize = 8;
+/// Default lookup→load depth (`loadq`): prefetch of load-sized batches.
+pub(crate) const LOAD_QUEUE_CAP_DEFAULT: usize = 14;
 /// Default load→scripts depth (`scriptq`): script is the long pole; modest buffer.
 pub(crate) const SCRIPT_QUEUE_CAP_DEFAULT: usize = 4;
 /// Default scripts→write depth: write is bursty (class_a head / tip flush); buffer
 /// script output so script thr does not stall on a full writeq.
-pub(crate) const WRITE_QUEUE_CAP_DEFAULT: usize = 20;
+pub(crate) const WRITE_QUEUE_CAP_DEFAULT: usize = 14;
 
 /// One load-sized run produced by lookup (decoded wire + pres, height-ordered).
 pub(crate) struct LoadBatch {
@@ -430,9 +430,9 @@ pub(crate) struct ConfirmQueueCaps {
 ///
 /// | Queue | Default |
 /// |-------|---------|
-/// | lookup→load (`loadq`) | **8** |
+/// | lookup→load (`loadq`) | **14** |
 /// | load→scripts (`scriptq`) | **4** |
-/// | scripts→write (`writeq`) | **20** |
+/// | scripts→write (`writeq`) | **14** |
 ///
 /// Env overrides removed (Q-04): change defaults in code if needed.
 #[inline]

crates/rbitcoin-net/src/ibd/confirm/tests.rs

@@ -298,7 +298,7 @@ fn split_wave_into_load_batches_is_eight_by_8000() {
         split_wave_into_load_batches, CONFIRM_BATCH_INPUTS_DEFAULT, CONFIRM_RUN_MAX_BLOCKS,
         LOAD_QUEUE_CAP_DEFAULT,
     };
-    assert_eq!(LOAD_QUEUE_CAP_DEFAULT, 8);
+    assert_eq!(LOAD_QUEUE_CAP_DEFAULT, 14);
     assert_eq!(super::confirm_queue_caps().load, LOAD_QUEUE_CAP_DEFAULT);
     assert_eq!(super::load_queue_cap(), LOAD_QUEUE_CAP_DEFAULT);
     assert!(super::LoadBatch { items: vec![] }.items.is_empty());

crates/rbitcoin-net/src/ibd/perf_log.rs

@@ -2134,7 +2134,7 @@ mod tests {
         s.arch_write_body_ms = 7;
         s.arch_write_head_ms = 2;
         let line = format_info(&s);
-        assert!(line.contains("loadq<0/8 scriptq=1/2 writeq=2/2"), "{line}");
+        assert!(line.contains("loadq<0/14 scriptq=1/2 writeq=2/2"), "{line}");
         assert!(line.contains("thru=200"), "{line}");
         // pin_residency slot always 0 (process pin FIFO removed); pin_plan_cache label retired.
         assert!(!line.contains("pin_res="), "{line}");
@@ -2452,7 +2452,7 @@ mod tests {
         assert!(!line.contains(" disk="), "{line}");
         // Depth 0 → `<` (consumer waiting on empty queue).
         assert!(
-            line.contains("loadq<0/8 scriptq<0/2 writeq=1/2") || line.contains("loadq="),
+            line.contains("loadq<0/14 scriptq<0/2 writeq=1/2") || line.contains("loadq="),
             "{line}"
         );
         assert!(line.contains("thru=200"), "{line}");
@@ -2574,7 +2574,7 @@ mod tests {
         assert!(!line.contains("cache="), "{line}");
         assert!(!line.contains("outfifo"), "{line}");
         assert!(!line.contains("sticky_fk="), "{line}");
-        assert!(line.contains("loadq=3/8 blks=8 wire=2MiB"), "{line}");
+        assert!(line.contains("loadq=3/14 blks=8 wire=2MiB"), "{line}");
         assert!(line.contains("scriptq=2/5 blks=16 wire=12MiB"), "{line}");
         assert!(
             line.contains("writeq=1/5 blks=16 wire=4MiB parents=500"),

crates/rbitcoin-query/src/combined_stage.rs

@@ -467,6 +467,31 @@ mod tests {
         let _ = std::fs::remove_dir_all(dir);
     }
 
+    #[test]
+    fn unresolved_heights_walks_past_taken_bq_hole() {
+        use std::collections::HashSet;
+        let (dir, q) = temp_query();
+        for h in 10..20u32 {
+            q.block_queue_enqueue(h, [h as u8; 32], 1, b"w").unwrap();
+        }
+        for h in 10..=12u32 {
+            assert!(q.block_queue_take_raw(h).is_some());
+        }
+        q.set_lookup_taken_hi(Some(12));
+        let none = HashSet::new();
+        assert_eq!(
+            q.block_queue_unresolved_heights(10, &none, 8),
+            vec![13, 14, 15, 16, 17, 18, 19]
+        );
+        assert!(q.block_queue_take_raw(15).is_some());
+        assert_eq!(q.block_queue_unresolved_heights(10, &none, 8), vec![13, 14]);
+        assert_eq!(
+            q.block_queue_unresolved_heights(16, &none, 8),
+            vec![16, 17, 18, 19]
+        );
+        let _ = std::fs::remove_dir_all(dir);
+    }
+
     #[test]
     fn lookup_take_removes_bq_row_via_query() {
         let (dir, q) = temp_query();

crates/rbitcoin-query/src/lib.rs

@@ -1670,14 +1670,22 @@ impl Query {
     ///
     /// One queue lock. Lookup wave select must use this instead of
     /// `list_meta` + per-height `is_resolve_complete`.
+    ///
+    /// Heights `≤ lookup_taken_hi` are already on loadq (`take_raw` removed
+    /// the BQ row). They are not a fetch hole — start after that high-water.
+    /// A missing height *above* the high-water still stops the walk.
     pub fn block_queue_unresolved_heights(
         &self,
         path_lo: u32,
         skip: &HashSet<u32>,
         cap: usize,
     ) -> Vec<u32> {
+        let start = match self.lookup_taken_hi() {
+            Some(hi) => path_lo.max(hi.saturating_add(1)),
+            None => path_lo,
+        };
         let g = self.block_queue.lock().unwrap();
-        g.unresolved_heights(path_lo, skip, cap)
+        g.unresolved_heights(start, skip, cap)
     }
 
     /// Load all queued blocks **with full payloads** (tests / tools).

docs/concurrency.md

@@ -15,7 +15,7 @@ Short map of who may write which tables. **Format is unstable until 1.0.**
 
 **IoSession TLS:** one completion session per OS thread (`with_thread_local`). Nested calls panic. Harvest tracks pending `(kind, epoch, slot)`; unexpected CQE / undrained leftover / CQ overflow is `Corrupt`, not a TipOnly miss. Drain before SQE buffers drop (spend annotate `DrainOnDrop`). Backends: Linux `io_uring`, portable `pool` (Darwin default), Windows IOCP. `RBITCOIN_IO=pread` disables the session.
 
-**Height-ordered unified pipeline (current):** peer → **body queue** (raw only) → **lookup** (in-order from `path_lo`, stop at first hole; decode + TipOnly `head_fk`; **dequeue** raw into `loadq=8`) → **load** (recv load-sized batch; stamp + pin + assemble) → scripts → write. Stage IO: [`invariants.md`](./invariants.md). **No** peer→confirm-feed wire retain. **No** hash-only / Class-A-only confirm (bq wire required). Load bind order is in-flight → published `live_union` chain → TipOnly (fence-connected). Lookup owns `live_union`; one `ArcSwap` of the layer-chain head per wave; get walks layers (no union rebuild); a layer stays while any height in its span is on the BQ **or** `tip − hi < recent_creates_horizon` (`2×soft_win`); disconnect stores None. Write drain inserts `tx.head` in parallel with Class C. Drain complete is max inserted **fk** (not tip/fence — those advance during drain). Header-cache GC polls store tip every load pack. In-flight prune is **after pin + scripts handoff**: drain-fk **and** `covers_fk_span` (TipOnly home — `docs/invariants.md`). No leftover pending map. Bodies without a known height are marked missing and re-getdata after the height map is ready — there is **no** dual-track archive-job / ContigPark fallback. Load pack **waits** on `feed.cv` when tip+1 is in `ready` but the BQ is not resolve-complete (no retain/BQ spin). Pack takes the feed mutex to collect candidates and again to mark inflight; one BQ `pack_snapshot` in between.
+**Height-ordered unified pipeline (current):** peer → **body queue** (raw only) → **lookup** (in-order from `max(path_lo, lookup_taken_hi+1)`, stop at first *fetch* hole; decode + TipOnly `head_fk`; **dequeue** raw into `loadq=14`) → **load** (recv load-sized batch; stamp + pin + assemble) → scripts → write. Stage IO: [`invariants.md`](./invariants.md). **No** peer→confirm-feed wire retain. **No** hash-only / Class-A-only confirm (bq wire required). Load bind order is in-flight → published `live_union` chain → TipOnly (fence-connected). Lookup owns `live_union`; one `ArcSwap` of the layer-chain head per wave; get walks layers (no union rebuild); a layer stays while any height in its span is on the BQ **or** `tip − hi < recent_creates_horizon` (`2×soft_win`); disconnect stores None. Write drain inserts `tx.head` in parallel with Class C. Drain complete is max inserted **fk** (not tip/fence — those advance during drain). Header-cache GC polls store tip every load pack. In-flight prune is **after pin + scripts handoff**: drain-fk **and** `covers_fk_span` (TipOnly home — `docs/invariants.md`). No leftover pending map. Bodies without a known height are marked missing and re-getdata after the height map is ready — there is **no** dual-track archive-job / ContigPark fallback. Load pack **waits** on `feed.cv` when tip+1 is in `ready` but the BQ is not resolve-complete (no retain/BQ spin). Pack takes the feed mutex to collect candidates and again to mark inflight; one BQ `pack_snapshot` in between.
 
 **Load claim pack size:** soft **Σ `tx.input`** budget (hardcoded **8000**; include overshoot block) or hard **144** blocks. Dense mainnet blocks hit the input soft stop after **typically a few blocks** (often 1–3); early tiny blocks may pack many until the hard cap. Do **not** treat ~32 as pack size (that was 8000/250 mid-chain, not fat-era).
 

docs/experimental-mainnet.md

@@ -124,7 +124,7 @@ peers than a dual-stack Core node (experimental user-agent still limits inbound)
 | `tx.head` seal | Segment roll builds fuse8 on seal (~27 M keys); watch seal begin/done logs — not a mono-head shadow fill |
 | Peer scarcity | [`OPERATOR.md`](../OPERATOR.md) § P2P transport (`x809` seeds + `P2P_V2` gossip). Experimental user-agent still limits inbound |
 | Mempool | Libre policy (0.1 sat/vB, full RBF + pure RBFR 1.25×, no dust ban, Libre annex); cluster **64 / 101 kvB** (Core-class); **scripts verified on accept** |
-| Confirm lookup/load | **Load** recvs load-sized batches (soft **8000** inputs / hard **144** blocks) from `loadq=8`. Dense mainnet is typically **a few blocks per batch**. IBD **lookup** TipOnly-resolves at most **64000** inputs or **1080** BQ-ready heights per wave, in order from `path_lo`. Real queues loadq=8 · scriptq=4 · writeq=20 |
+| Confirm lookup/load | **Load** recvs load-sized batches (soft **8000** inputs / hard **144** blocks) from `loadq=14`. Dense mainnet is typically **a few blocks per batch**. IBD **lookup** TipOnly-resolves at most **64000** inputs or **1080** BQ-ready heights per wave, in order from `path_lo`. Real queues loadq=14 · scriptq=4 · writeq=14 |
 | Not Core/Fulcrum | No production SLA; schema unstable until 1.0; reindex on incompatible layout changes |
 
 ## Related docs