[time-attack]

Efficiency wave 2 (GBRA-70), following #6364. Reads stop writing to pages, the two persistence reads a resident gbrain serve repeats stop scanning every write receipt ever stored, and a source-scoped vector search on a small source skips the index walk. Every number below comes from synthetic brains; no real brain data was used.

What changed

Retrieval tracking moves to page_retrievals (v224). The per-read UPDATE pages SET last_retrieved_at hit an indexed column, so it was never HOT. Each one wrote a new wide tuple plus entries in all 22 pages indexes and fired 7 triggers. Its statement trigger also advanced page_generation_clock_seq even when the 5-minute throttle matched no row, which invalidated the query-cache bookmark, the get_health memo and the projection-readiness memo on every read. The bump is now a pre-filtered upsert into a narrow side table:

• It has no foreign key, because the KEY SHARE lock would dirty the pages row. A statement trigger deletes the rows of hard-deleted pages instead.
• Readers (domain-bank samplers, volunteerUsageStats, entity card) take GREATEST(pages.last_retrieved_at, page_retrievals.last_retrieved_at). The migration copies the column, keeps its values and drops its unused index CONCURRENTLY.

Serve-loop persistence indexes (v223).

• The movement watermark read filtered v222's index on all committed receipts by incarnation and kind. On a worktree whose receipts aren't managed_sync_*, that meant a full walk that detoasted every intent.
• Receipts are permanent by design (compaction drops intent after 30 days), so the walk grew with every write. That is why it scaled with brain size, and retention itself is working as intended.
• The new partial index persistence_requests_sync_watermark replaces v222's index.
• compactWriteReceipts' candidate scan was a sequential scan, and persistence_requests_compactable replaces it.
• Both indexes are built CONCURRENTLY through buildIndexOnline.

Small-source vector scope.

• Both engines read pages.source_id statistics from pg_stats, cached for 60 s and only for source-scoped searches.
• Below a 10% page share, buildVectorSearchStatement omits indexWalkSql. The joined, exact and hasMore statements stay byte-identical, and missing statistics keep the walk.

Numbers

All runs: 4 vCPU / 16 GiB, Postgres 16.15 + pgvector 0.8.7, warm and prewarmed. Brains are 5,001 pages / 25,331 chunks and 50,010 pages / 248,802 chunks. Values are p50 ms, with pgbench N in brackets.

	5k before	5k after	50k before	50k after
Movement watermark read, largest worktree [25]	97	0.48	971	0.52
Receipt compaction scan [25]	2.6	0.65	30	0.28
Retrieval bump, 20 random pages [300]	5.8 (p95 16.8)	0.76 (1.6)	15.4 (27.9)	0.67 (1.6)
WAL from those 300 bumps	65 MB	0.8 MB	159 MB	1.2 MB
Generation clock advances from those 300 bumps	300	0	300	0
Vector search, remote reader, 0.1% source [25, 2 passes]	24–31	10–17	33–41	10–19
Vector search, remote reader, 1.7% source	42–49	19–29	87–88	67–76

A mixed serve workload ran for 150 s over stdio MCP (get_page ×4, search, list_pages, query, put_page per cycle, with pg_stat_statements):

	5k before	5k after	50k before	50k after
DB time	22.8 s	21.7 s	42.1 s	36.1 s
WAL	11 MB	6.4 MB	153 MB	15 MB
last_retrieved_at UPDATE share of WAL	53%	(0.5 MB upsert)	96%	(0.8 MB upsert)

Tool p50/p95 didn't move, since the bump is fire-and-forget.

Migration runtime at 50k: v223's watermark build took 2.3 s and the compactable build 0.07 s. v224's copy took 0.18 s. A full initSchema from 222 to 224 took 9 s.

Measured and not shipped

• SET STORAGE MAIN on content_chunks.embedding. At 5k it makes the joined plans pick HNSW in every stats state, but it's a net loss.
  • At 50k, keyword search went 230 → 330–350 ms and a content_chunks seq scan 40 → 470–500 ms. The walk didn't get faster.
  • Moving the rows needs a datum-changing rewrite: SET embedding = embedding keeps the TOAST pointer, so it has to be subvector(). At 50k that took 629 s plus a 146 s VACUUM, wrote 7.7 GB of WAL with 0 HOT updates, and left the TOAST space stranded.
• ef_search sizing. Any ef below 2× the window loses recall at both scales. Cutting only the walk LIMIT to 1.25–1.5× keeps recall and saves 3–7 ms at 50k, but hidden-row tolerance drops from 50% to 20–33%, and the remote-reader fixture would fall back.
• Outside this wave, on master too: at 50k, sources holding 4–16% of pages take 4.5–6.1 s and come back short. The joined statement's escalations and exact fallback hit the 8 s deadline.

Tests

• test/page-retrievals.test.ts runs the legacy UPDATE and the upsert over the same pages and ages:
  • same bumped set, same effective timestamps under the 5-minute throttle;
  • no clock advance (the legacy UPDATE advanced it even when matching nothing);
  • hard-delete cleanup and soft-delete retention;
  • every reader takes the newer of column and table.
• test/sync-movement.test.ts checks the watermark read gives identical results with and without index scans, across non-sync kinds, another incarnation, queued requests and compaction.
• test/serve-loop-migrations.test.ts and test/e2e/serve-loop-migrations-postgres.test.ts check a fresh install and an upgrade from 222: index swap, every timestamp copied, column untouched, idempotent re-run.
• test/e2e/persistence-consumers-postgres.test.ts checks the watermark plan uses the new index.
• test/search/vector-statement.test.ts, vector-index-walk.test.ts and the vector-plan E2E check that a small scope skips the walk and returns the joined rows, and that large scopes still walk. A forced probe with threshold 0 makes all three fail.
• Goldens were regenerated on purpose (catalog, migrations, exports types).

[trunk-io]

😎 This pull request was merged.