[time-attack]

Query-plan wave from the Efficiency crew (GBRA-67), based on GBRA-66's profile. I reproduced every claim on synthetic brains before fixing it. Brain data stays off this PR: the fixtures are synthetic, generated with the shape of a 5k-page brain. The bench harness (#6357) is not included.

Setup: 4 vCPU / 16 GiB, Postgres 16.15 + pgvector 0.8.7 (Docker), Bun 1.4.2. Brains are 5,001 pages / 25,331 chunks and 50,010 pages / 248,802 chunks, with 1024-dim vectors clustered by page type. All runs are warm, N=25, times in ms as p50/p95. Recall is measured against exactSql (exact scan).

1. Vector search walks the HNSW index first

What reproduced. The joined candidate statement's plan sits on a cost knife edge, for three reasons:

• 1024-dim vectors are TOASTed, so a sequential scan looks cheap.
• pgvector's HNSW estimate grows with hnsw.ef_search (the window).
• The private-page rule's correlated subplans price every page lookup high.

On the 5k brain it flips to a parallel seq scan plus an on-disk sort at 50 results. For remote readers it filters every page up front and nested-loops the chunk scan against that list.

Fix. indexWalkSql (ann) orders content_chunks alone with only the chunk-level filters, over twice the window. It then joins pages and sources by key and applies every page filter, visibility rule and the private-page rule, keeping the nearest innerLimit. It runs with enable_sort = off, so the HNSW scan is the only ordered path, and with ef_search sized for the window. A full window that fills the limit is the answer. Anything else falls back to the unchanged joined statement and its escalation path. Type and date filters skip the walk.

EXPLAIN ANALYZE, 5k brain, remote reader, limit 50:

before (joined)                                              after (walk)
Sort (actual 86.9..97.4)                                     Sort (actual 25.6)
  CTE hnsw_candidates                                          CTE ann
    Gather Merge (rows=250)                                      Limit (rows=500)
      Sort  external merge  Disk: 5872kB (+2 workers)              Index Scan using idx_chunks_embedding (actual 7.9..9.2)
        Parallel Seq Scan on content_chunks (8444 x 3)         CTE hnsw_candidates
      Index Scan using pages_pkey (loops=151)                    Nested Loop: CTE Scan ann -> content_chunks_pkey -> pages_pkey (loops=500)
Execution Time: 96-101 ms                                    Execution Time: 22-26 ms

At 50k the planner already uses the index, so the joined statement runs in 45 ms and the walk in 56 ms.

Engine searchVector (warm, p50/p95, recall in parentheses):

case	5k master	5k this PR	50k master	50k this PR
local, 20	177/658 (0.90)	24/49 (0.97)	23/37 (0.53)	31/43 (0.67)
local, 50	193/202 (1.00)	37/66 (0.995)	42/76 (0.63)	57/86 (0.79)
remote, 20	25/28 (0.90)	25/31 (0.97)	34/49 (0.53)	40/58 (0.67)
remote, 50	93/103 (1.00)	33/48 (0.995)	52/70 (0.63)	58/85 (0.79)
remote, 2 granted sources	42/92 (0.92)	27/31 (0.97)	27/46 (0.52)	43/70 (0.64)
remote, small source (0.1%)	12/20 (1.00)	40/47 (1.00)	20/24 (1.00)	44/64 (1.00)
remote, type filter (walk skipped)	239/377 (1.00)	256/421 (1.00)	8091/8110 (0.04, 2.8 of 20 rows)	1064/1157 (0.96)

At 50k the trade is honest: the walk costs 6 to 15 ms more at p50 because it reads twice the window, and recall rises 14 to 16 points. A selective source filter pays about 25 ms for the short walk before falling back, with identical results. The type-filter row's 50k win comes from fix 2.

PGLite 5k shares the statement builder, so I measured it too (p50/p95):

case	before	after
local, 20	131/789	17/24
local, 50	663/684	29/40
remote, 20	18/23	20/25
remote, 2 sources	50/70	21/25
small source	13/17	30/35
type filter	623/911	226/279

2. Private-page rule no longer trips JIT

What reproduced. The declared-lineage arm used slug IN (SELECT ... jsonb_array_elements_text(...)), which the planner prices as a 100-row semi-join per page. Every read carrying the rule crossed jit_above_cost, so Postgres compiled it on every call. list_pages spent 37 to 46 ms on a statement that runs in under 1 ms. The backlink count in search and query was worse: 265 ms at 5k and 571 ms at 50k, nearly all of it JIT (pg_stat_statements jit_functions 20k to 23k per 84 calls).

Fix. The arm is now slug = ANY(ARRAY(SELECT ...)). It returns the same rows, the estimated cost per page drops, and no statement needs a JIT-off wrapper. After the change, no statement in the hot-path suite shows jit_functions > 0.

statement / tool	5k master	5k this PR	50k master	50k this PR
list_pages statement, remote rule (direct)	38.6/43.1	3.1/5.2	38.3/68.3	1.9/2.3
list_pages over MCP (warm stdio)	70.4/78.7	11.3/15.9	52.1/79.5	11.5/15.3
backlink count statement (mean, pg_stat_statements)	264.7	4.0	570.6	7.9
search over MCP (with Voyage query embedding)	880/993	668/747	866/1023	715/841
recall over MCP	662/807	569/617	1385/2954	664/1098

3. Held-page read joins pages by key

readHeldPages (every get_page and every search result set) joined on p.id::text = page_id, so pages could only be scanned. Whether that cost anything depended on small-table statistics. Right after import, it seq-scanned pages once per source: 17 ms at 5k in EXPLAIN ANALYZE, and a 12.8 to 13.9 ms mean in pg_stat_statements during the warm MCP run. Once sources and op_checkpoints were analyzed, master's planner skipped the probe and ran in about 1 ms, so I could not reproduce the slow case at 50k. The join is now p.id = (page_id)::int, a primary-key probe in every statistics state (0.5 to 1.2 ms). MCP get_page at 50k measured 25.3/40.3 before and 12.8/18.3 after.

I also grepped for the same id::text join pattern. The other hits are tiny tables (sources, persistence_worktrees), a doctor-only join and a one-time migration, so I left them alone.

4. get_page looks up the exact slug first (GBRA-66 #4)

An alias-resolving readPageSnapshot (get_page, fetch) used one statement with slug = $1 OR EXISTS (alias), which scans the source's pages. It now runs the exact-slug statement first and the alias statement only on a miss. The alias statement's ORDER BY (p.slug = $1) DESC already ranks any exact match above every alias match, so a hit returns the same row, and resolved_slug is unchanged. requireUnambiguous (the put_page identity path) still runs the alias statement alone, because its count includes alias matches.

	5k master	5k this PR	50k master	50k this PR
snapshot statement (mean, pg_stat_statements)	3.77	0.71	1.79	0.78
get_page over MCP, warm (includes fix 3)	18.1/29.0	12.1/18.7	14.6/22.0	9.4/15.5

At 50k I measured 1.8 ms, not GBRA-66's 23–80 ms; the plan depends on statistics, as in fix 3.

Not shipped: expansion overlap (GBRA-66 #10)

I implemented it: the expansion call and the original query's embed and vector search start before the lexical arms, and a test proved identical results and meta. I measured no gain, though, so I took it out of this PR. In-process A/B on Postgres with real Voyage embeddings and a fixed 800 ms expansion, N=25: 1378/1565 with the overlap against 1365/1462 without at 5k, and 1396/1635 against 1424/1777 at 50k. Over MCP with the real LLM (N=25, two rounds each), p50 was 2209/2268 with the overlap against 2154/2203 without. The traces show the overlap does start the original embed alongside the expansion, but the post-expansion stages took longer by about the same amount. The patch is parked for a follow-up.

Flake fix

test/bounded-child-exec.test.ts failed once in the Ubicloud gate. The SIGTERM test slept 200 ms and expected the child's lockfile to exist; on a loaded runner the child had not started. The child now installs its trap before writing the lockfile, and the test waits for the lockfile. Forced probe: delaying the child by 0.5 s fails the old test and passes the new one.

Tests (results unchanged)

• test/search/vector-index-walk.test.ts (PGLite, new) runs both statements with index scans disabled, so each computes its window exactly. It asserts a full walk window returns the joined statement's rows row-for-row, order and pool counts included. The filter matrix covers private-page rules, archived, quarantined and deleted pages, stale hashes, granted sources, slug exclusion and detail: low. It also checks that a selective source leaves the walk short and searchVector returns the joined result. A mutation that drops visibility from the walk fails 8 of 9 cases.
• test/search/vector-statement.test.ts pins the walk's shape: ann touches only content_chunks, page filters sit after the key joins, parameters are shared, and the walk is absent for the legacy guard, wide columns and type or date filters. The existing master statements are byte-identical across both dialects and every option combination.
• test/e2e/vector-plan-real-column-postgres.test.ts now explains the walk and asserts it uses idx_chunks_embedding for remote readers and 50- and 200-result windows.
• test/e2e/private-visibility-plan-postgres.test.ts (new) asserts both spellings hide the same pages, in the same order, across every rule. That covers explicit private, atoms with a private origin, synthesized concepts, derived_from as string, array and .md, missing and cross-source lineage, and life-chronicle events. It also checks the planner's cost per page is now under jit_above_cost / 500 (147 against 296), with a precondition that fails if the old spelling would pass.
• test/page-snapshot-exact-first.test.ts (new) covers exact hits (one statement, no alias lookup), alias fallback, an exact page in another source outranking an alias, deleted and private exact pages, the ambiguity check and a miss. The existing alias-redirect suite (test/get-page-federated-scope.test.ts) passes unchanged.
• The SQL-text and RLS-inventory goldens were regenerated on purpose. The private-page rule's text changed in every method that carries it, and searchVector now runs one more scoped read (the walk).
• test/held-read-signals.test.ts asserts the held-page read returns the hold and its revision, and that its plan probes pages_pkey. Forced probe: this fails on the text-cast join and passes after.

Not fixed here (bigger; in the GBRA-67 report)

• sync-movement writer-status query (persistence_requests count with intent->>'kind' LIKE): 132 ms at 5k and 836 ms at 50k per call during serve. It is not on the per-call read path.
• UPDATE pages SET last_retrieved_at on reads: last_retrieved_at is indexed, so these updates are never HOT.
• content_chunks.embedding storage: TOASTed vectors are why the seq-scan cost is underestimated. SET STORAGE MAIN plus a rewrite would make plans honest, but it changes the schema and needs a migration.

[trunk-io]

😎 This pull request was merged.