docs(db): Phase 5 close-out — measurement baselines, staging parity, and D4 settled to fact (#316)

docs/branch-review-records/da9eadddac4befebc995b1000caa73f1fa0ced0d25d52b5789223410bac06c08.record.md

@@ -0,0 +1 @@
+| 2026-08-21 | claude/phase-5-closeout | c3ca68fa7f13e2d7ae6c4c96ccfcafb67a59135a | Phase 5 close-out: measurement baselines, staging parity, D4 documentation correction (#316) | Self-review; no blocking findings. Docs-only diff plus five ledger inbox requests. Read-only production window, one staging-only db push, one live-drift dispatch (green). Two limits and one attribution finding recorded rather than absorbed; no RAG surface touched. | verify:pr-local (11/11, failed: none); check:production-readiness (READY, 9 PASS, 0 blocking); format (whole tree, committed); live-drift run 32514326022 green |

docs/branch-review-records/eecaf3f51b0f4d8dad88a76dbb9c162887d5067cd48bdde0bbaa87a410f4701a.record.md

@@ -0,0 +1 @@
+| 2026-08-21 | claude/phase-5-closeout | 653712cbeda0059979e71131e828241f921f17da | PR #2250 review-thread sweep: Codex P1/P2 + CodeRabbit findings on the Phase 5 close-out docs and ledger inbox | Fixed and resolved. P1 db-push contradiction removed (db push reserved for authorised staging/recovery); plan-flip and index-units deliverables left explicitly OPEN; never-reset claim qualified to database-wide only; Perth/UTC boundary made explicit; four MD040 fences labelled. Supersedes the c3ca68fa record, whose checks cell compressed the gate output. | verify:pr-local decisive line — "PR-local verification summary: - completed: check:runtime, check:installed-lock-parity, format:changed, sitemap:check, docs:check-index, docs:check-inventory, docs:check-scripts, docs:check-links, check:branch-review-ledger, check:outstanding-issues, check:ledger-write-discipline / - failed: (none) / - not reached: (none)"; format (whole tree, committed) |

docs/outstanding-issues-inbox/05bd88ac-f24f-464a-8853-5d7ff55b375b.json

@@ -0,0 +1,14 @@
+{
+  "version": 2,
+  "id": "05bd88ac-f24f-464a-8853-5d7ff55b375b",
+  "createdOn": "2026-08-21",
+  "action": "add",
+  "payload": {
+    "pri": "P2",
+    "type": "task",
+    "summary": "The document_index_units retrieval path has no EXPLAIN baseline, and Phase 5 has no query-specific plan-flip evidence",
+    "detail": "Two Phase 5.1 deliverables are explicitly OPEN, not discharged. Re-graded P3 -> P2 versus the withdrawn request 2040d1fb, because that request understated the gap by claiming substitute coverage that does not exist. (A) NO EXPLAIN BASELINE FOR THE INDEX-UNITS PATH. public.explain_retrieval_rpc accepts exactly four names -- match_documents_for_query, match_document_chunks_text, match_document_lookup_chunks_text, match_document_table_facts_text -- and raises 22023 Unsupported retrieval RPC for anything else, proven against production for both match_document_chunks_text_v2 and match_document_index_units_hybrid_v2. For the first of those the v1 sibling match_document_chunks_text shares the owning table document_chunks and is a usable stand-in. For the second there is none: match_document_index_units_hybrid_v2 delegates to match_document_index_units_hybrid_scoped over document_index_units (supabase/schema.sql:8033-8054), and no supported RPC touches that table. document_index_units is one of the two section 1.2 outliers, so the outlier that most needed a baseline is the one that has none. (B) NO QUERY-SPECIFIC PLAN-FLIP EVIDENCE. explain_retrieval_rpc EXPLAINs `select * from public.<rpc>(...)`, so a PL/pgSQL body's inner plan is never exposed and every sample reports a single Function Scan with no index names. Plan section 5.1's 'record plan flips (seq scan -> index scan)' is therefore unanswerable through this instrument. The pg_stat_user_indexes read captured in Phase 5.1(c) is a WEAKER and DIFFERENT signal, not a substitute: idx_scan is cumulative across every workload touching the table and no before/after delta was captured around the samples, so it can prove an index is never chosen by anything but cannot prove that a given query changed plan. NEXT: one migration extending the explain_retrieval_rpc p_rpc branch list to the _v2 family (at minimum match_document_index_units_hybrid_v2 and match_document_chunks_text_v2), shipped in an approved window -- with D4 ON, merging it to main deploys it, so it needs the window and a green post-merge live-drift run. Then re-run npm run profile:retrieval --analyze to capture the missing baseline. For (B), consider whether an auto_explain-style capture is a better fit than widening the RPC. STOP: do not record the cumulative index-usage read as plan-flip evidence; that conflation is exactly what this row exists to prevent.",
+    "source": "Codex review of PR #2250 (P2, comment 3833062803 and 3833062807); docs/audit/live-drift-forensics-2026-08.md Phase 5 close-out 5.1(b); supabase/schema.sql:8033-8054",
+    "issueUlid": "01M0JWT29F8VAY97CG3PM5X2QW"
+  }
+}

docs/outstanding-issues-inbox/0a0ab127-0cc1-4b14-b4ce-fd839a98386c.json

@@ -0,0 +1,12 @@
+{
+  "version": 2,
+  "id": "0a0ab127-0cc1-4b14-b4ce-fd839a98386c",
+  "createdOn": "2026-08-21",
+  "action": "update",
+  "payload": {
+    "id": "#231",
+    "detail": "PHASE 5.2 CONFIRMED SATISFIED with fresh data 2026-08-22 Perth (2026-08-21 UTC), not reopened. This row already recorded that remediation-plan Phase 5.2 is satisfied by S1's 2026-08-17 healthy-latency probes; the Phase 5 close-out re-measured production end to end and confirms it. Retrieval now costs 955 ms on the text fast path and 6,720 ms on hybrid (from 31,610 ms and 21,757 ms at the incident), against answerRouteBudgetMs.fast of 25,000 ms -- so retrieval consumes 4-27% of the fast budget and is no longer capable of binding it. The 2026-08-14 verdict that pre-generation latency WAS the binding cause stands for that window and is now closed out. Residual R4 (chronic ~30 s strong-route provider_timeout on metformin-renal-dosing and valproate-pregnancy, with a safe source-backed extractive fallback) is generation-side and unchanged; no separate R4 row was created, per this row's own instruction. INCONSISTENCY TO FIX AT RECONCILE: the recommended-execution-queue row for #231 still presents it as A1 / 'immediate approved live investigation' with the old framing ('live answers degrade to source-only when answerRouteBudgetMs.fast binds while retrieval is healthy'), which contradicts the P1 -> P2 re-grade recorded in this detail row. The queue entry should be re-graded to match P2 and re-scoped to the R4 generation-side residual, so the queue stops advertising a retrieval investigation that the measurements have closed.",
+    "source": "docs/audit/live-drift-forensics-2026-08.md Phase 5 close-out 5.1(a) and 5.2; production probes 2026-08-22 Perth (2026-08-21 UTC)",
+    "baseRowFingerprint": "a2d8deb8060b77d1ed3718818a01b31372c4b30281165dc049102fa616b396e9"
+  }
+}

docs/outstanding-issues-inbox/1c89922d-3d01-4811-a255-ed78e2ed11c3.json

@@ -0,0 +1,10 @@
+{
+  "version": 2,
+  "id": "1c89922d-3d01-4811-a255-ed78e2ed11c3",
+  "createdOn": "2026-08-21",
+  "action": "cancel",
+  "payload": {
+    "requestId": "2040d1fb-6d26-4977-902f-d4a3c2c404c2",
+    "reason": "Withdrawn and replaced: the request asserted that the v1 siblings profiled in Phase 5 cover the same owning tables as the two unreachable _v2 RPCs. That is false for one of the two. match_document_index_units_hybrid_v2 delegates to match_document_index_units_hybrid_scoped over document_index_units, and none of the four RPCs explain_retrieval_rpc supports touches that table, so the index-units retrieval path has no EXPLAIN baseline at all rather than a partial stand-in. Raised by Codex review of PR #2250 and confirmed against supabase/schema.sql:8033-8054. Superseded by a corrected add request queued in the same commit."
+  }
+}

docs/outstanding-issues-inbox/2040d1fb-6d26-4977-902f-d4a3c2c404c2.json

@@ -0,0 +1,14 @@
+{
+  "version": 2,
+  "id": "2040d1fb-6d26-4977-902f-d4a3c2c404c2",
+  "createdOn": "2026-08-21",
+  "action": "add",
+  "payload": {
+    "pri": "P3",
+    "type": "task",
+    "summary": "explain_retrieval_rpc cannot profile the _v2 retrieval RPC family, so the two Phase 1.2 outliers have no EXPLAIN baseline",
+    "detail": "Phase 5 close-out was asked to capture an EXPLAIN baseline for match_document_chunks_text_v2 and match_document_index_units_hybrid_v2 (the two section 1.2 outliers) and could not. public.explain_retrieval_rpc accepts exactly four names -- match_documents_for_query, match_document_chunks_text, match_document_lookup_chunks_text, match_document_table_facts_text -- and raises 22023 Unsupported retrieval RPC for anything else. Verified three ways: its only defining migration 20260626020000_phase7_retrieval_rpc_performance.sql, the supabase/schema.sql live mirror, and a direct call against production returning the 22023 error for both names. Extending the p_rpc branch list is production DDL, and with D4 confirmed ON a migration merged to main deploys automatically, so it needs its own approved window; it was deliberately NOT written during a read-only phase. SECOND, SMALLER GAP recorded at the same time: the function EXPLAINs `select * from public.<rpc>(...)`, so a PL/pgSQL body's inner plan is never exposed and every sample reports a single Function Scan with no index names. Plan section 5.1's 'record plan flips (seq scan -> index scan)' is therefore not answerable through this instrument at all; pg_stat_user_indexes is the durable substitute and is what Phase 5 used. NEXT: author one migration extending the branch list to the _v2 family and ship it in an approved window. STOP: not urgent -- the v1 siblings profiled in Phase 5 cover the same owning tables.",
+    "source": "docs/audit/live-drift-forensics-2026-08.md Phase 5 close-out 5.1(b); supabase/migrations/20260626020000_phase7_retrieval_rpc_performance.sql",
+    "issueUlid": "01M0JTGY6SJYJEM6MAR4FGVWA8"
+  }
+}

docs/outstanding-issues-inbox/3c71ce2a-b752-4f1b-a8ef-4870f8027bf7.json

@@ -0,0 +1,14 @@
+{
+  "version": 2,
+  "id": "3c71ce2a-b752-4f1b-a8ef-4870f8027bf7",
+  "createdOn": "2026-08-21",
+  "action": "add",
+  "payload": {
+    "pri": "P2",
+    "type": "issue",
+    "summary": "Restored trigram indexes have never been scanned, so the 2026-08-14 incident fix is probably mis-attributed to them rather than to ANALYZE",
+    "detail": "Measured on production 2026-08-22 Perth (2026-08-21 UTC), read-only. pg_stat_database.stats_reset is NULL, so no database-wide reset has been recorded -- note this does NOT prove the counters were never reset, because pg_stat_reset_single_table_counters(oid) resets one table or index without touching that database-level timestamp; 163 of 210 public indexes show scans, max(idx_scan) 262,335,345, total 344,747,815 -- the counters are demonstrably live. Against that, ALL 22 indexes restored or repaired by the incident response report idx_scan = 0: the 20 Phase 4 rebuilds of 2026-08-19, and both trigram indexes restored 2026-08-14 -- document_chunks_content_trgm_idx (68 MB, OID 1491258) and documents_title_trgm_idx (648 kB, OID 1491257). The OID matters: 1491258 is the same value Phase 4 Step 4 recorded as UNCHANGED across the 2026-08-19 push, so the index was never dropped and recreated and its counter has accumulated for the full seven days since restoration. What actually serves retrieval on those tables is document_chunks_search_idx (GIN tsvector, 37,717 scans), document_chunks_embedding_hnsw_idx (HNSW, 4,906) and documents_title_search_idx (37,299). Note that a DIFFERENT trigram index on a sibling table, document_table_facts_title_row_param_trgm_idx, is used 2,372 times, so this is not a blanket property of the planner or of pg_trgm. CONSEQUENCE: forensics Phase 5 attributes the 31,610 -> 1,535 ms recovery to 'restore + ANALYZE', treating the trigram restore as the fix. Zero scans mean those indexes are very unlikely to be carrying that recovery -- unlikely rather than impossible, because a per-relation counter reset would be invisible to the read above; the OID continuity across the 2026-08-19 push is what makes the zero credible, and ANALYZE was an unseparated co-intervention in the same operation -- 20260626020000 itself ends by running analyze over documents, document_chunks, document_table_facts, document_embedding_fields and document_index_units. Stale planner statistics is the better-supported explanation for both the 31-second regression and its recovery -- better-supported, not established. THIS DOES NOT RETRACT THE REPAIR: the two indexes are repo-defined, the migration chain commands them, and drift is correctly closed by their presence. It retracts the ATTRIBUTION, and it changes triage order -- the next 30-second retrieval incident should check planner statistics BEFORE hunting missing indexes. Secondary and non-urgent: roughly 69 MB of currently unscanned index. NEXT: (a) in an approved window, first rule out a per-index counter reset -- on PostgreSQL 18+ read pg_stat_all_indexes.stats_reset for these two OIDs, and on earlier versions fall back to whatever reset or audit record exists (the production server version was not read in this window); (b) separate the two interventions by measuring retrieval latency before and after a fresh ANALYZE; and (c) only then decide whether the two trigram indexes earn their storage. STOP: do NOT drop them -- they are repo-defined and dropping them reopens live drift.",
+    "source": "production read-only window 2026-08-22 Perth (2026-08-21 UTC); docs/audit/live-drift-forensics-2026-08.md Phase 5 close-out 5.1(c)",
+    "issueUlid": "01M0JTGYEJ47M1XD0V0EGS5BDD"
+  }
+}

docs/outstanding-issues-inbox/98e6ae7f-4190-49f3-a64d-fbf632ef6643.json

@@ -0,0 +1,11 @@
+{
+  "version": 2,
+  "id": "98e6ae7f-4190-49f3-a64d-fbf632ef6643",
+  "createdOn": "2026-08-21",
+  "action": "done",
+  "payload": {
+    "id": "#316",
+    "outcome": "CLOSED 2026-08-22 Perth (2026-08-21 UTC) -- Phase 5, this row's last remaining follow-on, is complete and the alarm is OBSERVED clear rather than merely believed clear. (1) MEASUREMENT, like-for-like with the section 1.3 probe style: text fast path 31,610 -> 955 ms (33x better than the incident, 1.6x better than the 2026-08-14 post-restore reading); hybrid 21,757 -> 6,720 ms. Nothing was worse than the 2026-08-17 healthy baseline, so nothing was escalated. A new EXPLAIN forward baseline was captured on the healthy system for all four RPCs that explain_retrieval_rpc supports, zero errors across twelve samples; the new slowest path is match_document_table_facts_text (1.72 s unprimed, 1,070 unprimed shared_read). Section 1.3's two pending items are now closed: missing-index sizing is moot (0 missing) and the rag_retrieval_logs miss-scan baseline is recorded (47 seq scans, 6,620 tuples read, 640 live rows, 1.4 MB -- cheap at this scale; the row to watch as it grows is rag_aliases at 5,829 seq scans over 68 rows). (2) 5.2 confirmed ALREADY SATISFIED via #231 and not reopened: retrieval now consumes 4-27% of the 25 s fast-route budget, so it cannot bind it. (3) 5.3 needed NO eval canary -- Phase 3 deployed zero behaviour change -- so it reduced to check:production-readiness: READY, 9 PASS, 0 blocking failures. (4) PRODUCTION RE-VERIFIED UNCHANGED, 8/8 against the 2026-08-21 baseline: 210 public indexes, 0 invalid, 211 history rows, 20 no-statement rows, migration_history_versions() probe ok with 211 versions, search_schema_health() ok true. (5) STAGING PARITY CLOSED: 20260820120000 applied to ikoiolksxqxfxgiyqpnu by real db push (never migration repair); staging now reads 211 rows with the function present and stmt_count 3, matching production exactly. (6) THE ALARM IS OBSERVED CLEAR: live-drift run 32514326022, dispatched on main, is GREEN -- 'No unexpected schema drift between live and supabase/schema.sql', 'Remote migration versions: 211 (read via rpc)', 'Remote-only (Preview blockers): 0', every step success. That is the FIRST green live-drift since 2026-07-19T19:37:29Z (of the last 60 runs only four succeeded), ending a 33-day red streak, and pinned issue #1963 auto-closed at 2026-08-21T18:38:42Z. (7) D4 DOCUMENTATION CORRECTED: this row's 2026-08-19 'auto-deploy is OFF' sentence was already superseded by the 2026-08-21 correction; the coordination board and the forensics dossier are now aligned to the dashboard fact (Deploy to production ENABLED, production branch main, automatic branching ON limit 3), with the superseded safe-either-way reasoning retained and marked rather than deleted. THREE FINDINGS QUEUED SEPARATELY rather than absorbed here: the uncapped Branching Compute cost; the explain_retrieval_rpc gap on the _v2 RPC family (needs production DDL, deliberately not written in a read-only phase); and the discovery that all 22 restored indexes report idx_scan = 0 with no database-wide statistics reset recorded, which makes ANALYZE rather than the trigram restore the better-supported explanation for the incident recovery (a per-relation pg_stat_reset_single_table_counters() call would not show in that read, so this is well-supported, not proven). RESIDUAL WORK LIVES ON ITS OWN ROWS, not here: PITR is still OFF on production, and the fifteen no-statements history rows remain #Q5JHBJ.",
+    "baseRowFingerprint": "17b4358a583c8bf5d2b7cc77a6ccf0b8eaf714b92a4e7301f3756a9e831b46ad"
+  }
+}

docs/outstanding-issues-inbox/d873ec0d-e25d-41fe-b24c-d2c3aa9375ee.json

@@ -0,0 +1,14 @@
+{
+  "version": 2,
+  "id": "d873ec0d-e25d-41fe-b24c-d2c3aa9375ee",
+  "createdOn": "2026-08-21",
+  "action": "add",
+  "payload": {
+    "pri": "P2",
+    "type": "rec",
+    "summary": "Supabase preview-branch compute is an uncapped cost sitting outside the organisation Spend Cap",
+    "detail": "Dashboard read 2026-08-21 (the same read that settled D4) shows Automatic branching ON with limit 3 and \"Supabase changes only\" enabled, and the same screen warns that Branching Compute is NOT covered by the organisation's Spend Cap. Preview databases are therefore an uncapped cost that scales with the number of open PRs touching supabase/**, on a project that already runs auto-deploy to production. CONTEXT FOR ANY DECISION TO REDUCE OR DISABLE THEM: CI's Migration replay job (db-reset-verify, which runs `supabase migration up --local`) independently replays the entire migration chain on every database-touching PR, so preview branches are a SECOND net rather than the only one -- turning them down would not leave the chain unproven. Preview branches did earn their keep once (the 20260819100200 guard failure on PR #2151 was caught by a preview branch building from the chain alone), so this is a cost/benefit decision, not a cleanup. NEXT: decide whether to keep branching at limit 3, lower the limit, or disable it, and record the decision with its date. STOP: do not change Supabase project settings without explicit owner approval; this row is a decision request, not a licence to reconfigure the project.",
+    "source": "Supabase dashboard read 2026-08-21; docs/audit/live-drift-forensics-2026-08.md D4 section; AGENTS.md Supabase project safety",
+    "issueUlid": "01M0JTGY049X40BT8EZTS24YYX"
+  }
+}