[Hades32sBot]

Problem

SqlitePersistence::stream_document_log ignores page_size and eagerly materializes the entire queried timestamp range into a Vec before returning a stream.

The worst instance: PersistenceReader::max_ts() calls load_documents(TimestampRange::all(), Order::Desc, page_size = 1) during Database::load — so on every boot, the backend parses every row of the documents table (all live docs plus all MVCC revisions and tombstones) into ConvexValues, twice (once via new_idle_repeatable_ts(), once via DatabaseSnapshot::max_ts). On a self-hosted instance with a few hundred MB of document history this allocates ~1.7 GB of heap and dominates startup time. The same eager collection applies to the retention worker's expired_documents scans, which is the OOM pattern reported in #358.

Change

Fetch rows in pages of page_size instead:

• load_docs gains a LIMIT clause and a keyset-pagination clause over the (ts, table_id, id) primary key ((ts, table_id, id) > / < (?, ?, ?) matching the ORDER BY direction). When resuming, the redundant range bound on the ordering side is omitted so SQLite seeks into the PK index instead of re-scanning preceding rows on every page; for tablet-filtered scans +table_id = ? keeps the planner on the PK index.
• stream_document_log_pages (new #[try_stream] method) loops: run one LIMIT page_size query per page (prepare_cached), hold only that page in memory, and release the Mutex<Connection> between pages — which also lets writers interleave during long scans.
• retention_validator.validate_document_snapshot(min_ts) runs once upfront and after each page, since retention may advance while the lock is released between pages.
• load_documents / load_documents_from_table pass page_size through; page_size.max(1) guards against a 0 page size.

page_size remains a pure fetch-chunk-size hint per the trait docs, so stream contents and ordering are unchanged. Note the stream is no longer a single-statement snapshot across pages — matching the Postgres/MySQL drivers' contract.

Impact

• max_ts() at boot now reads 1 row instead of the whole table: startup memory is roughly O(page_size) instead of O(documents JSON size), which measured ~1.7 GB on a real self-hosted DB.
• Retention catch-up over large backlogs no longer spikes by the full remaining range, and each page query is O(page_size) rather than quadratic in the range size.

Testing

cargo check/cargo clippy/rustfmt --check clean for -p sqlite. No sqlite-persistence unit tests exist in the repo; the pagination logic was verified against a ~1.2M-row production-shaped DB copy and exercised exhaustively (all orders/filters/page sizes/ranges incl. page_size=0) against a brute-force reference.

cc @Hades32

[cubic-dev-ai]

All reported issues were addressed across 1 file

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[Hades32sBot]

@goffrie could you have a look?