diff --git a/docs/audit/live-drift-forensics-2026-08.md b/docs/audit/live-drift-forensics-2026-08.md index 0a85055d45..713c58ca40 100644 --- a/docs/audit/live-drift-forensics-2026-08.md +++ b/docs/audit/live-drift-forensics-2026-08.md @@ -1236,6 +1236,434 @@ batch, deliberately not bundled into this docs-only PR because migrations are an surface with their own replay gates. The other 19 drift findings, the 2 unexpected live indexes, and the green live-drift dispatch also remain **pending** for the full phase. +### Phase 4 completion — 2026-08-19 (owner-authorised off-peak production window) + +_Owner-authorised window against `Clinical KB Database` (`sjrfecxgysukkwxsowpy`) for index DDL plus +a `supabase db push` of the guard migrations. Executed from a dedicated worktree; the main checkout +`D:\Repos\Database` stayed linked to STAGING throughout. D4 is OFF (the Supabase GitHub auto-deploy +was disabled before this window), so nothing in this task reached production on merge — every hosted +change below was made by the explicit step that names it._ + +**Tooling substitution, recorded.** The Supabase MCP connector was blocked by this session's +permission classifier, so every hosted statement went through the authenticated Supabase CLI 2.114.0 +(`supabase db query --file` / `db push`), which reaches the same management API. Two transport traps +cost a retry each and neither touched the database: Node cannot `execFile` the `supabase` npm shim on +Windows (`ENOENT` — resolve `node_modules/supabase/dist/supabase.js` and run it with `node`), and +`db query` parses a leading `--` as a flag, so SQL beginning with a comment must be passed via +`--file`. + +#### Step 1 — restore point: PITR is NOT enabled (deviation, stated not absorbed) + +`supabase backups list --project-ref sjrfecxgysukkwxsowpy` reports `"pitr_enabled": false` with +`"walg_enabled": true` and seven retained daily physical backups, the most recent `COMPLETED` at +**2026-08-17T20:33:28Z** — roughly 38 hours before this window. **No PITR restore point exists to +confirm.** The window proceeded on the explicit assessment that every statement in it is index-only +with an exact one-statement inverse (`CREATE INDEX CONCURRENTLY` ↔ `DROP INDEX CONCURRENTLY`) and no +data-loss surface — the same reasoning the 2026-08-14 incident window recorded. **This is a real gap +in the plan's safety model, not a cleared checklist item:** the plan's standing rule "PITR/backup +restore point captured before any mutating phase" cannot be satisfied on this project as configured, +and any future phase that mutates _data_ rather than indexes must not proceed on this precedent. +Enabling PITR is an owner dashboard decision. + +#### Step 2 — pre-flight, then the twenty builds + +Read-only pre-flight (`db query --linked --project-ref sjrfecxgysukkwxsowpy`), matching §3.7 exactly: + +``` +db postgres · usr postgres · total_rows 199 · latest_version 20260818113000 · documents 2851 +``` + +Of the 24 indexes in scope, exactly **4** were present: the two 2026-08-14 trigram restores +(`documents_title_trgm_idx`, `document_chunks_content_trgm_idx`, both `indisvalid`/`indisready`) and +the two `unexpected_live` orphans. All **20** `missing_live` indexes were confirmed absent — the §1.3 +inventory still held at the window. + +Owning-table sizes at repair time (the §1.3 sizing debt, now discharged; heap only): + +| Table | Heap | `n_live_tup` | Batch | +| -------------------------------- | ---------- | -----------: | ----------- | +| `document_index_units` | 162 MB | 113,587 | B | +| `document_chunks` | 124 MB | 70,120 | B | +| `document_table_facts` | 48 MB | 34,795 | (drop only) | +| `document_images` | 19 MB | 14,267 | B | +| `image_caption_cache` | 10224 kB | 3 | A | +| `document_summaries` | 4272 kB | 2,851 | A | +| `documents` | 3928 kB | 2,851 | B | +| `document_index_quality` | 2760 kB | 2,851 | A | +| `ingestion_job_stages` | 1888 kB | 7,979 | A | +| `medication_records` | 792 kB | 656 | A | +| `rag_queries` | 552 kB | 373 | A | +| `indexing_v3_agent_jobs` | 272 kB | 2,065 | A | +| `rag_query_misses` | 128 kB | 177 | A | +| `rag_aliases` | 32 kB | 68 | A | +| `api_rate_limits` | 8192 bytes | 4 | A | +| `audit_logs` | 0 bytes | 0 | A | +| `storage_cleanup_jobs` | 0 bytes | 0 | A | +| `document_publication_approvals` | 0 bytes | 0 | A | + +Each build ran `CREATE INDEX CONCURRENTLY IF NOT EXISTS` with the canonical definition, then re-read +`pg_index.indisvalid`/`indisready` and compared normalised `pg_get_indexdef` against the canonical +text using the repo's own `normalizeIndexDefinition` (`tests/supabase-schema.test.ts:199`). Canonical +text came from `supabase/drift-manifest.json` `snapshot.indexes[].def` — the rendered form of each +defining migration — and every one was cross-read against that migration's own `create index` +statement before the window (`20260712165211` ×14, plus `20260717170000`, `20260717131000`, +`20260705010000`, `20260708000000`, `20260705230000`, `20260608001000`). **No transactional build was +ever attempted.** + +**Batch A — small tables, 14/14 OK** (all `indisvalid=true indisready=true`, definitions matched): + +``` +audit_logs_action_created_idx OK size=8192 bytes +audit_logs_owner_created_idx OK size=8192 bytes +api_rate_limits_bucket_updated_idx OK size=16 kB +rag_aliases_type_enabled_idx OK size=16 kB +rag_queries_source_chunk_ids_gin_idx OK size=56 kB +rag_query_misses_aliases_idx OK size=16 kB +image_caption_cache_owner_hash_idx OK size=1392 kB +document_index_quality_owner_score_idx OK size=152 kB +document_publication_approvals_document_idx OK size=8192 bytes +document_summaries_owner_idx OK size=104 kB +indexing_v3_agent_jobs_locked_at_idx OK size=8192 bytes +ingestion_job_stages_job_stage_started_idx OK size=616 kB +medication_records_owner_category_idx OK size=32 kB +storage_cleanup_jobs_owner_status_idx OK size=8192 bytes +``` + +**Batch B — large tables, 6/6 OK**, built one at a time in ascending owning-table size with a +`pg_locks` reading between each. Baseline `waiting 0 · total_locks 9 · active_backends 0`; after every +build `waiting 0`, never above `total_locks 9`. **No lock contention at any point.** + +``` +documents_registry_projection_lookup_idx OK size=72 kB (documents) +document_images_hash_idx OK size=1616 kB (document_images) +document_images_structured_profile_gin_idx OK size=64 kB (document_images) +document_images_visual_intelligence_version_idx OK size=16 kB (document_images) +document_chunks_anchor_idx OK size=1288 kB (document_chunks) +document_index_units_heading_path_idx OK size=4104 kB (document_index_units) +``` + +**Zero invalid builds, zero retries, zero skips** — the drop-and-retry-once path and the +skip-and-report path were both defined and neither was needed. `#102`'s bare-column indexes on +`documents` were held out entirely and remain its own canary-gated work. + +`ANALYZE` was then run on all eighteen touched tables (three of the twenty are expression indexes, +which gather statistics on the expression only at `ANALYZE`). + +#### Step 3 — the two `unexpected_live` indexes: DROP, not codify + +Both are strict leading-column subsets of a present, valid canonical index, and in both cases the repo +chain already **commands the drop** — codifying either would contradict a committed migration: + +| Orphan | Superseded by (present, valid on live) | Repo instruction | +| -------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `document_table_facts_document_id_idx` `(document_id)`, 296 kB | `document_table_facts_document_idx (document_id, page_number)`, 560 kB | created by `20260618000000:10`, dropped by `20260620000000:159` with the comment "superseded by document_table_facts_document_idx(document_id, page_number)"; `20260712172000` drops it again | +| `storage_cleanup_jobs_owner_id_idx` `(owner_id)`, 8192 bytes | `storage_cleanup_jobs_owner_status_idx (owner_id, status, created_at DESC)` — restored in Batch A above | dropped by `20260703030000:40` ("`storage_cleanup_jobs_owner_id_idx` -> `storage_cleanup_jobs_owner_status_idx`") and again by `20260708000000:26` | + +Neither name appears in `supabase/schema.sql` or the manifest, so dropping them moves live **into** +agreement with the mirror and needs no new migration. Both dropped with `DROP INDEX CONCURRENTLY IF +EXISTS`; a follow-up `pg_class` read returns zero rows for both. The `storage_cleanup_jobs` drop was +deliberately ordered **after** its superseding composite was built. + +**Whole-schema result:** `pg_indexes` in `public` now reports **210** indexes against the manifest's +**210** repo-defined (192 − 2 + 20 = 210), all 22 guard targets `indisvalid AND indisready`, and +**zero** invalid-or-not-ready indexes anywhere in the schema. + +#### Step 4 — codification + +Four migrations, all authored to the `20260804110240` pattern where they are guards (`set local` +timeouts, validate presence + `indisvalid`/`indisready` + normalised `pg_get_indexdef`, never build, +exactly one `raise exception`): + +| Migration | Validates | +| ------------------------------------------------------ | ---------------------------------------------------------------------------------------------- | +| `20260819100000_restore_batch_a_operational_indexes` | the 14 Batch A indexes | +| `20260819100100_restore_batch_b_retrieval_indexes` | the 6 Batch B indexes | +| `20260819100200_restore_search_health_trigram_indexes` | `documents_title_trgm_idx` + `document_chunks_content_trgm_idx` — **plan 4.4 debt discharged** | +| `20260819100300_monitor_restored_retrieval_indexes` | (not a guard) redefines `search_schema_health()` `required_indexes` | + +`20260819100200` closes the hole §1.1 named explicitly: `20260804110240` validates four _other_ +indexes and never checked this pair, which is why both could vanish between 2026-07-05 and 2026-08-02 +while the chain still replayed green. It resolves canonical names only — an `index_aliases` entry +satisfying the health probe is not evidence the canonical trigram index exists, which is the exact +failure being guarded. + +All three guards were **dry-run against production before the push** (the DO block only reads and +raises) and all three passed. One attempt returned a transient Cloudflare `502` from +`api.supabase.com` and succeeded unchanged on retry — a transport failure, not a guard failure. + +The 20 index definitions are already in `supabase/schema.sql`, and validation-only guards create +nothing, so **no mirror change accompanies the three guards** — consistent with `20260804110240`, +whose DO block likewise does not appear in `schema.sql`. Only `search_schema_health()` was mirrored. +`npm run drift:manifest` → `Replay complete in 21s`, `Wrote supabase/drift-manifest.json`; the diff is +exactly one `def_hash` (`f4f5f536…` → `85df52de…`, `search_schema_health`) plus the regeneration +stamps. The index inventory is byte-identical, as it must be: no index was added to or removed from +`schema.sql`. + +**`required_indexes`: all eight Phase 6.3 monitor-candidates are now monitored.** The list grows from +22 to 30. Three of the eight (`documents_registry_projection_lookup_idx`, `document_chunks_anchor_idx`, +`document_index_units_heading_path_idx`) were among the twenty absent indexes and were rebuilt and +validated **before** this migration was written, so it cannot turn the probe red on a still-absent +object; the other five are present GIN indexes on the lexical half of the retrieval RPCs that had no +monitored equivalent (`document_index_units` was the worst-covered table in scope at 2 of 16). +`supabase/search-health-unmonitored-indexes.json` drops from 44 to 36 entries — the coverage test +rejects an entry that is also monitored — and now contains **no** `monitor-candidate`: every remaining +entry is a reasoned `accepted-unmonitored`. + +**`migration_history` allowlist: zero new entries, and that is a measured result, not an omission.** +The condition was to allowlist any of the fifteen `#Q5JHBJ` no-statements versions that is +index-shaped _and_ whose objects these guards now validate. Six of the fifteen are index-shaped; their +created objects were enumerated and intersected against the 22 this window's guards validate: + +| No-statements version | Index objects it creates | Covered by these guards | +| ---------------------------------------------------- | --------------------------------------------------------------------------- | ----------------------: | +| `20260702110000 drop_redundant_indexes` | (drops only) | — | +| `20260702150000 documents_owner_covering_index` | `documents_owner_id_covering_idx` | **0/1** | +| `20260702180000 promote_index_generation_id_columns` | six `*_document_generation_idx` | **0/6** | +| `20260712165915 reconcile_ingestion_index_shapes` | three `import_batches_*` / `ingestion_jobs_*` | **0/3** | +| `20260712170500 codify_live_operational_indexes` | 43 names incl. `audit_logs_owner_id_idx`, `document_summaries_owner_id_idx` | **0/43** | +| `20260712172000 drop_redundant_table_fact_indexes` | (drops only) | — | + +The intersection is **empty**. The near-misses are name-adjacent but distinct objects +(`audit_logs_owner_id_idx` ≠ `audit_logs_owner_created_idx`; `document_summaries_owner_id_idx` ≠ +`document_summaries_owner_idx`). No honest `validation` entry exists, so none was written and the +fifteen stay unallowlisted — the state Phase 6.2 predicts, with fail-fast guard migrations still the +fix. **Consequence: the `migration_history` finding count does NOT drop in this phase.** It stays at +15 unexpected + 5 expected. That remains `#Q5JHBJ`'s work. + +#### Step 5 — production push (real execution, no `migration repair`) + +Performed only after all 22 indexes a guard validates were confirmed built. The `supabase migration +list` pre-flight showed the pending set was **exactly** the four new versions and nothing else — itself +confirmation that D4 auto-deploy is off, since none had been applied by merge. + +``` +$ supabase db push --linked --project-ref sjrfecxgysukkwxsowpy --skip-vault --yes +Applying migration 20260819100000_restore_batch_a_operational_indexes.sql... +Applying migration 20260819100100_restore_batch_b_retrieval_indexes.sql... +Applying migration 20260819100200_restore_search_health_trigram_indexes.sql... +Applying migration 20260819100300_monitor_restored_retrieval_indexes.sql... +{"upToDate":false,"dryRun":false,"migrations":[...4 files...],"seeds":[],"roles":[],"message":"Finished supabase db push."} +``` + +`--skip-vault` kept the push to migrations only; no vault secret was read or written. `supabase +migration list` after: **pending 0**, every local version matched remotely. `migration repair --status +applied` was never used. + +History rows carry executed statements — the CLI's per-statement shape, **not** the empty mark-applied +shape — so the guard-migration contract is not breached and none of these four will ever surface in +the `migration_history` probe: + +``` +20260819100000 restore_batch_a_operational_indexes stmt_count 4 no_statements false +20260819100100 restore_batch_b_retrieval_indexes stmt_count 4 no_statements false +20260819100200 restore_search_health_trigram_indexes stmt_count 4 no_statements false +20260819100300 monitor_restored_retrieval_indexes stmt_count 4 no_statements false +``` + +`search_schema_health()` on production against the expanded 30-index list: + +``` +ok true · missing [] · legacy_ivfflat_indexes [] +``` + +#### Step 6 — staging brought to parity (ref re-verified before every call) + +Target `ikoiolksxqxfxgiyqpnu` re-verified before each step by an identity read; the corpus check +(`documents = 0`, versus production's 2,851) was the abort condition and was re-run every time, in +code, not by eye. Production was never a target in this step. + +`document_chunks_content_trgm_idx` carried the 2026-06-06 form §3.3(d) predicted: + +``` +before ... gin (lower(((COALESCE(section_heading, ''::text) || ' '::text) || content)) gin_trgm_ops) +after ... gin (lower(((COALESCE(section_heading, ''::text) || ' '::text) || COALESCE(content, ''::text))) gin_trgm_ops) +``` + +Dropped and rebuilt concurrently into the canonical `coalesce(content, '')` form +(`20260705180000:11`), `indisvalid`/`indisready` both true — now identical to production and the +manifest (`8499c3d3…`). This had to precede `20260819100200`, which validates that exact form. + +The four migrations were then applied by the §2.2/§2.5 Phase 2 method — the repository file's content +run verbatim, then an explicit history row carrying the repository's own version and name. +`apply_migration` was not used (it stamps a connector-generated version, which +`docs/staging-setup.md` forbids); `db push` was not used either, so staging's one-element `statements` +shape stays consistent with its other 29 such rows. Faithfulness read back from staging: + +``` +20260819100000 · restore_batch_a_operational_indexes stmt_count 1 · bytes 7616 · md5 05f64e164882b7ba813cc67c93cbadcc · matches repo file true +20260819100100 · restore_batch_b_retrieval_indexes stmt_count 1 · bytes 6233 · md5 85a9268cf193fdee1b38bf75fd7d2181 · matches repo file true +20260819100200 · restore_search_health_trigram_indexes stmt_count 1 · bytes 5571 · md5 bbae64185d2a0271b9c4ca18c40680e3 · matches repo file true +20260819100300 · monitor_restored_retrieval_indexes stmt_count 1 · bytes 10836 · md5 13619a6b83458f17a62b6c0130e73ae9 · matches repo file true +``` + +After: `total_rows 203 · latest_version 20260819100300 · no_statements 0`, `documents 0`, +`document_chunks 0` (corpus untouched), `search_schema_health() ok true · missing []`. + +**`check:drift` against staging — GREEN, zero unexpected drift** (was **19** at §2.3 and still 19 at +the §2.5 re-measure). `npm run check:drift` itself could not authenticate in this session — it +resolves its target from `NEXT_PUBLIC_SUPABASE_URL`/`SUPABASE_SERVICE_ROLE_KEY` via +`createAdminClient()`, and reading `.env.local` was blocked — so the comparison was run through the +**same exported `compareDriftSnapshots()`**, the same manifest, the same allowlist and the same +manifest-staleness pre-check, with only the transport changed (staging's `schema_drift_snapshot()` +fetched over the authenticated CLI). `--prune-stale` was **not** used, per the §2.4 trap: + +``` +Target: staging ikoiolksxqxfxgiyqpnu · documents 0 · migrations 203 +Drift manifest: generated 2026-08-18T18:15:50.121Z from schema.sql 328677d1c6f3… +Compared 6 extensions, 38 tables, 1 views, 93 functions, 210 indexes, 48 policies, 170 constraints, 26 triggers, 2 storage_buckets against live. + +Stale allowlist entries (5) — no longer matching: + ? [migration_history] no_statements 20260701010000 + ? [migration_history] no_statements 20260701020000 + ? [migration_history] no_statements 20260701030000 + ? [migration_history] no_statements 20260701060000 + ? [migration_history] no_statements 20260702000000 + +No unexpected drift. +EXIT=0 +``` + +The five stale entries are production's seeded `superseded` guards reading stale against staging — the +documented §2.4 warning-only condition, deliberately not pruned. Staging is now at **full parity with +the repository chain**: the §2.3 finding set — (a) `work_mem`, (b) eight never-created objects, +(c) chain-stale columns, (d) the trigram definition — is completely closed. + +#### Step 7 — live-drift proof: 37 → 16 findings, indexes fully closed + +Dispatched on `main` (head `4666708b2`, before this branch merged): **Actions run +[`32171070287`](https://github.com/BigSimmo/Database/actions/runs/32171070287)**, 2026-08-18T18:27:03Z. + +``` +Drift manifest: generated 2026-08-18T08:30:22.062Z from schema.sql 87ac9fc4849e… +Compared 6 extensions, 38 tables, 1 views, 93 functions, 210 indexes, 48 policies, 170 constraints, 26 triggers, 2 storage_buckets against live. + ~ [migration_history] no_statements … ×5 (the five seeded superseded guards — expected) +UNEXPECTED DRIFT (16): + ! [functions] mismatch public.search_schema_health() :: def_hash: manifest="f4f5f536026c4dd27d506a8e40b8c6d7" live="85df52de66e4e89d4a328b81a3a87c90" + ! [migration_history] no_statements … ×15 (20260701040000, 20260702100000…180000, 20260712165915…173000) +``` + +| Category | §3.7 (run `32131517648`) | This run (`32171070287`) | Verdict | +| ------------------------- | -----------------------: | -----------------------: | ------------------------------- | +| `missing_live` indexes | **20** | **0** | **closed** | +| `unexpected_live` indexes | **2** | **0** | **closed** | +| function `def_hash` | 0 | 1 | expected — merge-pending, below | +| `migration_history` | 15 (+5 seen) | 15 (+5 seen) | unchanged — `#Q5JHBJ`'s work | +| **Total unexpected** | **37** | **16** | | + +**Zero `missing_live`, zero `unexpected_live` — the two targets this phase owned.** + +**The one function mismatch is this branch not yet being on `main`, proven rather than assumed.** The +run compares `main`'s manifest, generated 2026-08-18T08:30 from `schema.sql 87ac9fc4849e…`, against +live. Live now reports `85df52de66e4e89d4a328b81a3a87c90` — **byte-identical to the `def_hash` in this +branch's regenerated `drift-manifest.json`**, which is the only `def_hash` that changed. So live +matches the repo _as of this branch_; the finding is a repo-behind-live artefact of dispatching before +merge and clears when this PR lands. It is not a new divergence: no `match_*` RPC mismatched, and the +RPC track closed in §3.7 stays closed. + +**`migration_history` did not drop, as Step 4 predicted.** No guard here validates any object created +by those fifteen versions, so no allowlist entry was earned. Unchanged is the correct outcome, not a +shortfall. + +**Phase 4 status: complete.** Plan 4.1 (Batch A), 4.2 (Batch B), 4.3 (unexpected disposition), 4.4 +(guard migrations + `schema.sql` mirror + regenerated manifest + `required_indexes`) and 4.5 (green +index proof) are all discharged. Remaining `#316` work is the `migration_history` block, which is +`#Q5JHBJ`, and Phase 5's after-measurements. + +#### Step 8 — the guard caught a real chain defect on the Supabase preview branch + +The `20260819100200` trigram guard **failed CI on PR #2151**, and it was right to. The Supabase +Preview check (an ephemeral preview branch database, project `jgzqdaalxnfmiadmpnib` — neither +production nor staging) builds from the migration chain alone and reported: + +``` +ERROR: The retrieval-critical trigram indexes restored on 2026-08-14 are not present in canonical +form; ... Missing: (none); Invalid: (none); Mismatched: document_chunks_content_trgm_idx (SQLSTATE P0001) +At statement: 3 +``` + +**Root cause — the first creator wins, and every later one is a no-op.** Three renderings of this +index exist in the repository and the chain permanently produces the oldest: + +| Migration | Expression | Effect on a fresh replay | +| ------------------- | ------------------------------------------------------------------------ | ------------------------------ | +| `20260606000000:11` | `lower(coalesce(section_heading,'') \|\| ' ' \|\| content)` | **creates it — this one wins** | +| `20260622000000:13` | `lower(coalesce(section_heading,'') \|\| ' ' \|\| coalesce(content,''))` | `if not exists` → **no-op** | +| `20260705180000:11` | identical to `20260622000000` = `schema.sql:743` = **canonical** | `if not exists` → **no-op** | + +`grep -c "drop index.*document_chunks_content_trgm_idx" supabase/migrations/` returns **zero** — no +migration ever drops it, so the two correct definitions can never take effect. Any database built +from migrations alone therefore carries the 2026-06-06 form while `schema.sql`, the drift manifest +and production carry the `coalesce(content,'')` form. The difference is not cosmetic: the older +expression evaluates to NULL for any row with NULL `content`, so those chunks are absent from the +trigram index entirely. + +**This was already visible and was mis-scoped as staging-only.** §3.3(d) found exactly this and +recorded it as "a chain-stale residual on staging only", repaired by hand in the staging window. It +is not staging-only — it is every environment built from the chain: `supabase db reset`, a +disaster-recovery replay, CI's `Migration replay` job, and the preview branch. The hand-repair fixed +the symptom on one database; the chain kept producing the wrong index. The Phase 4.4 guard is what +turned a silent, environment-specific divergence into a loud, reproducible CI failure — which is +precisely the behaviour the guard-migration contract exists to buy. + +**Fix: `20260819100150_reconcile_chain_stale_content_trgm_index.sql`**, ordered between the Batch B +guard (`100100`) and the trigram guard (`100200`) so a fresh replay is canonical before it is +validated. It is deliberately conditional, and will never run a write-blocking index build on a +populated hosted database: + +| Situation | Behaviour | +| ------------------------------- | -------------------------------------------------------------------------------- | +| already canonical | early `return` — no lock, no DDL (production and staging today) | +| wrong form, table **empty** | `drop index` + `create index` in canonical form (preview, `db reset`, DR replay) | +| wrong form, table **populated** | `raise exception` telling the operator to rebuild concurrently out of band first | + +**Proof, run locally against the same scratch Postgres image the manifest generator uses, replaying +the whole chain in order (the local stand-in for CI's `Migration replay` and the preview branch):** + +``` +# with the fix removed — reproduces the CI failure exactly +FAILED at 20260819100200_restore_search_health_trigram_indexes.sql: +ERROR: The retrieval-critical trigram indexes ... Mismatched: document_chunks_content_trgm_idx +Applied 201/203. + +# with the fix in place +Applied 204/204. +document_chunks_content_trgm_idx after full chain replay: + CREATE INDEX document_chunks_content_trgm_idx ON public.document_chunks USING gin (lower(((COALESCE(section_heading, ''::text) || ' '::text) || COALESCE(content, ''::text))) gin_trgm_ops) +RESULT: CANONICAL — matches schema.sql / manifest / production +``` + +**No-op path proven separately**, because this migration must eventually run against a populated +production table. Re-running it on an already-canonical database left the index **OID unchanged** +(`18657` → `18657` in the scratch replay), meaning no rebuild and no lock, and the `100200` guard +still passed afterwards. + +**Applied to both hosted tiers, and the no-op verified on production itself.** The CLI refused the +first push with `LegacyDbPushMissingRemoteError` — "Found local migration files to be inserted +before the last migration on remote database" — because `100150` sorts before the already-applied +`100200`/`100300`. That is the documented out-of-order case and its documented flag; the pending set +was confirmed to be exactly this one file before using it: + +``` +$ supabase db push --linked --project-ref sjrfecxgysukkwxsowpy --skip-vault --include-all --yes +Applying migration 20260819100150_reconcile_chain_stale_content_trgm_index.sql... +``` + +Production `document_chunks_content_trgm_idx` **OID `1491258` before and `1491258` after**, identical +`pg_get_indexdef`, `search_schema_health() ok true` — the early-return branch, confirmed on the real +70,120-row table rather than inferred. Staging took the same migration by the §2.2 Phase 2 method +(`md5 aa2d6edef30a1ef73924c74c0a9216a3`, matches the repo file), reaching **204** history rows with +`no_statements 0` and its corpus untouched; the staging drift comparison is still **green, zero +unexpected drift**. + +`schema.sql` and `drift-manifest.json` are deliberately **unchanged** by this fix: the mirror already +declared the canonical form, and it was the chain that disagreed with it. Nothing to re-mirror, and +the manifest sha still matches. + +**Ordering note for future sessions.** `20260819100150` is intentionally out of order relative to +`100200`/`100300`, which were applied first. Any future `supabase db push` that legitimately needs to +insert a version before the remote tip must pass `--include-all`, and must confirm the pending set +first — the flag applies _every_ locally-absent version, not just the intended one. + ## Phase 5 — Measure and close the loop _Partially run 2026-08-14 (incident scope); full close-out still requires the remaining phases._ diff --git a/docs/branch-review-records/1fd0204537f6955d37613b686bc81de075429ecd88cce458fd455865c870a467.record.md b/docs/branch-review-records/1fd0204537f6955d37613b686bc81de075429ecd88cce458fd455865c870a467.record.md new file mode 100644 index 0000000000..81e872df65 --- /dev/null +++ b/docs/branch-review-records/1fd0204537f6955d37613b686bc81de075429ecd88cce458fd455865c870a467.record.md @@ -0,0 +1 @@ +| 2026-08-18 | claude/db-remediation-phase4-indexes-a1661a | 5d3dca4dd7a0593a35e6d639144092dee6609256 | Phase 4 index restoration: 20 concurrent index builds + 2 concurrent drops on production, 3 fail-fast guard migrations, search_schema_health required_indexes 22->30, schema.sql mirror, regenerated drift manifest, staging parity, forensics evidence | PASS — 20/20 indexes rebuilt indisvalid+indisready with canonical definitions, 2 orphans dropped per the repo chain, live-drift 32171070287 shows missing_live 20->0 and unexpected_live 2->0, staging drift green (was 19). Two escalations recorded not absorbed: PITR is not enabled on production, and no migration_history allowlist entry was earned (empty intersection with the 15 no-statements versions) | check:migration-role; vitest supabase-schema + search-health-index-coverage + migration-history-guards + drift-detection + migration-history-placeholders + hosted-migration-role-guard (6 files, 109 tests); drift:manifest; check:rag:fixtures (36 golden cases); check:medication-interactions; check:medication-lexicon-report; verify:pr-local all stages pass except two load-induced timeouts (codex-cloud-setup, document-viewer-page-virtualization) that pass in isolation and are unrelated to this diff | diff --git a/docs/database-remediation-coordination.md b/docs/database-remediation-coordination.md index 7c8af78af1..52697981a0 100644 --- a/docs/database-remediation-coordination.md +++ b/docs/database-remediation-coordination.md @@ -79,24 +79,24 @@ _Board re-baselined 2026-08-17 from `main` `f5b093291`; **updated 2026-08-18 fro forensics file, and the PR list — worker self-reports were checked against merged content, not trusted as-is._ -| Phase | Status | PR / evidence | Notes | -| ----------------------------------------------- | ------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Prereq: docs on `main` | ☑ done | #1906 (plan/playbook), #2027 (traps), #2044 (this board, `e663203ce`) | — | -| Prereq: `SUPABASE_ACCESS_TOKEN` secret (`#183`) | ☐ pending — not blocking | repo-level secret names checked 2026-08-17: not present | Owner chose the Supabase MCP connector for all windows (2026-08-17). Token gates CLI repair paths only. | -| 0 — Enablement | ☑ COMPLETE 2026-08-14 | #1938, #1939, #1951 (proof run `31813064485`, auto-issue #1963), #1978 | Issue #1963 stays open until Phases 3–4 land; now that Phase 6.1 is on `main` repo-side, weekly live-drift will _additionally_ report `schema_drift_snapshot()` as repo-ahead plus an info line "migration-history probe not present" — expected, not a regression. | -| 1.1 — History fingerprint | ☑ done (owner action open) | #1960 §1.1 | `20260705180000` NOT mark-applied (14 stmts). Drop window 2026-07-05→08-02; Phase 2 narrowed it: staging **kept** the four indexes production lost, so an out-of-band drop on production is the leading explanation. Dashboard audit-history pairing = owner. `#248` open. | -| 1.2 — RPC divergence dossier | ☑ COMPLETE 2026-08-18 | **#2087** merged `173ea9f28` → forensics §1.2; `#316` row updated by reconcile #2098 (`d113396b0`) | Read-only connector window, 4 SELECTs, zero writes. **All 10 mismatches are attribute-only `SET work_mem`** present on live, absent from `schema.sql`; strip it → manifest hash 10/10. 4 mirror-stale (64 MB = `20260724000000`), 6 live-ahead attribute-only (128 MB ×4 hybrids; live-only 64 MB on `chunks_text_v2` and on `table_facts_text` after its `20260724120000` recreate). Zero body divergences, zero repo-ahead, zero UNCLASSIFIED. | -| 1.3 — Inventory / EXPLAIN | ◐ whole-schema done; sizing pending | #1970 §1.3 | 20 absent, 2 orphaned, zero invalid. Remaining index sizing + `rag_retrieval_logs` miss-scan baseline pending — fold into Phase 5. | -| 2 — Staging parity (`#056`) | ☑ COMPLETE 2026-08-18 (re-measured) | **#2093** merged `e546840a1` (§2.0–2.4); **#2104** merged `f19cf8f60` (§2.5 re-measure); `#056` update `5ed2f873` pending reconcile | Gap was 28. Staging `ikoiolksxqxfxgiyqpnu` replayed to parity, corpus untouched, production never targeted. `check:drift` vs staging RED 19 at base `ed43a64f2`; **re-measured at current `main` after applying `20260818090000`: still exactly 19** (same keys, same hashes). Snapshot v2 `migration_history` block runs end-to-end on staging (probe `ok`, 0 rows). Findings: (a) 7 `work_mem` fn mismatches; (b) 8 objects `schema.sql` declares that no migration creates; (c) 3 column-set mismatches; (d) `document_chunks_content_trgm_idx` def mismatch. All four dispositioned by Phase 3. §2.4 traps: staging KEPT the indexes production lost; four duplicate migration version pairs; `20260725000000` points staging's worker GUC at production — **never seed `cron_ingestion_jwt` / `indexing_v3_agent_secret` / `ingestion_webhook_secret` into staging's vault**; **never run `check:drift --prune-stale` against staging** (the five seeded `migration_history` allowlist entries are production's and read stale there — warning only). | -| 3 — Reconciliation (reframed) | ☑ COMPLETE 2026-08-18 — repo-side, staging-proven, **and live on production** (auto-applied; see §3.7) | **#2106** merged `72aa18865` (migrations `20260818110000` work_mem ×10, `111000` eight schema-only objects, `112000` chain-stale columns; forensics §3.1–3.6); **#2111** merged `9b52eb075` (staging proof + `20260818113000` forward-codify of three hybrid RPC bodies the chain never received; `#316` final + `#183` rider + 2 captures + 3 cancels queued) | D1 codify-as-live, D2 exemption granted. Manifest `def_hash` == live production hash 10/10 — the ten RPC "divergences" are closed as a mirror gap. §3.3: `token_estimate` was chain-stale (added by `112000`, no-op on prod); `rag_visual_eval_*` id defaults rebound to `pg_catalog.gen_random_uuid()` (no-op on prod); (d) production's restored `document_chunks_content_trgm_idx` (`coalesce(content,'')` form, `8499c3d3…`) IS canonical — staging carries the 2026-06-06 form; repair = Phase 4.4 guard migration + staging drop-and-recreate. Staging at 199 migrations, single drift residual = that index. **New finding:** `20260712000000`'s owner step never landed for three hybrid RPCs — chain-stale, forward-codified by `113000` (not a production hole). Tooling: `scripts/check-drift.ts:192` clips column diffs to 240 chars (never names the column) — queued P3; PR template lacks `RAG impact:` guidance — queued P3. | -| 4 — Index restoration | ◐ PARTIAL (incident) — **NEXT — after D4 is decided** | #1960 §Phase 4 | 2 trgm restored 2026-08-14; **20** still missing, 2 unexpected undispositioned; **4.4 guard migration for the restored trgm pair owed** (`20260804110240` pattern) plus staging drop-and-recreate of `document_chunks_content_trgm_idx` (2026-06-06 form → canonical); decide the 8 Phase 6.3 monitor-candidates via a `required_indexes` migration. Opus, off-peak prod window. | -| 5 — Measure | ◐ PARTIAL (incident) | #1960 §Phase 5 | Text fast path 31,610 → 1,535 ms; hybrid 8,519 ms residual. Full EXPLAIN set, `work_mem` before/after (only if D1 changes values), `check:production-readiness` pending. | -| 6 — Future-proofing | ◐ REPO-SIDE COMPLETE 2026-08-18 | **#2058** merged `9c660af1f` (14 paths verified on `main`); follow-up #2090 merged `79e5d2a68` | 6.1 migration `20260818090000_schema_drift_snapshot_history_probe.sql` built + mirrored + manifest regenerated, **LIVE on production since 2026-08-18** (auto-applied on merge; §3.7 verified probe `ok`, 20 `migration_history` rows = 15 unallowlisted §1.1 + 5 seeded). Next for 6.2: fail-fast guard migrations for the 15 — never bare allowlisting. 6.2 contract + `supabase/drift-allowlist.json` (5 seeded superseded guards; ~18 §1.1 versions deliberately NOT allowlisted → first post-deploy run reports them; fix = fail-fast guard migrations, never bare allowlisting). 6.3 `tests/search-health-index-coverage.test.ts` + unmonitored list (44 entries, 8 monitor-candidates). New P2 row queued (inbox `d6ce8a1d`). Stale branch `claude/database-drift-allowlist-48839e` (tip `3fe1deafe`) was recreated after merge — redundant, owner may delete. | -| 7 — Deferred | ☐ not started | — | Each its own session, per playbook. | +| Phase | Status | PR / evidence | Notes | +| ----------------------------------------------- | ------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Prereq: docs on `main` | ☑ done | #1906 (plan/playbook), #2027 (traps), #2044 (this board, `e663203ce`) | — | +| Prereq: `SUPABASE_ACCESS_TOKEN` secret (`#183`) | ☐ pending — not blocking | repo-level secret names checked 2026-08-17: not present | Owner chose the Supabase MCP connector for all windows (2026-08-17). Token gates CLI repair paths only. | +| 0 — Enablement | ☑ COMPLETE 2026-08-14 | #1938, #1939, #1951 (proof run `31813064485`, auto-issue #1963), #1978 | Issue #1963 stays open until Phases 3–4 land; now that Phase 6.1 is on `main` repo-side, weekly live-drift will _additionally_ report `schema_drift_snapshot()` as repo-ahead plus an info line "migration-history probe not present" — expected, not a regression. | +| 1.1 — History fingerprint | ☑ done (owner action open) | #1960 §1.1 | `20260705180000` NOT mark-applied (14 stmts). Drop window 2026-07-05→08-02; Phase 2 narrowed it: staging **kept** the four indexes production lost, so an out-of-band drop on production is the leading explanation. Dashboard audit-history pairing = owner. `#248` open. | +| 1.2 — RPC divergence dossier | ☑ COMPLETE 2026-08-18 | **#2087** merged `173ea9f28` → forensics §1.2; `#316` row updated by reconcile #2098 (`d113396b0`) | Read-only connector window, 4 SELECTs, zero writes. **All 10 mismatches are attribute-only `SET work_mem`** present on live, absent from `schema.sql`; strip it → manifest hash 10/10. 4 mirror-stale (64 MB = `20260724000000`), 6 live-ahead attribute-only (128 MB ×4 hybrids; live-only 64 MB on `chunks_text_v2` and on `table_facts_text` after its `20260724120000` recreate). Zero body divergences, zero repo-ahead, zero UNCLASSIFIED. | +| 1.3 — Inventory / EXPLAIN | ◐ whole-schema done; sizing pending | #1970 §1.3 | 20 absent, 2 orphaned, zero invalid. Remaining index sizing + `rag_retrieval_logs` miss-scan baseline pending — fold into Phase 5. | +| 2 — Staging parity (`#056`) | ☑ COMPLETE 2026-08-18 (re-measured) | **#2093** merged `e546840a1` (§2.0–2.4); **#2104** merged `f19cf8f60` (§2.5 re-measure); `#056` update `5ed2f873` pending reconcile | Gap was 28. Staging `ikoiolksxqxfxgiyqpnu` replayed to parity, corpus untouched, production never targeted. `check:drift` vs staging RED 19 at base `ed43a64f2`; **re-measured at current `main` after applying `20260818090000`: still exactly 19** (same keys, same hashes). Snapshot v2 `migration_history` block runs end-to-end on staging (probe `ok`, 0 rows). Findings: (a) 7 `work_mem` fn mismatches; (b) 8 objects `schema.sql` declares that no migration creates; (c) 3 column-set mismatches; (d) `document_chunks_content_trgm_idx` def mismatch. All four dispositioned by Phase 3. §2.4 traps: staging KEPT the indexes production lost; four duplicate migration version pairs; `20260725000000` points staging's worker GUC at production — **never seed `cron_ingestion_jwt` / `indexing_v3_agent_secret` / `ingestion_webhook_secret` into staging's vault**; **never run `check:drift --prune-stale` against staging** (the five seeded `migration_history` allowlist entries are production's and read stale there — warning only). | +| 3 — Reconciliation (reframed) | ☑ COMPLETE 2026-08-18 — repo-side, staging-proven, **and live on production** (auto-applied; see §3.7) | **#2106** merged `72aa18865` (migrations `20260818110000` work_mem ×10, `111000` eight schema-only objects, `112000` chain-stale columns; forensics §3.1–3.6); **#2111** merged `9b52eb075` (staging proof + `20260818113000` forward-codify of three hybrid RPC bodies the chain never received; `#316` final + `#183` rider + 2 captures + 3 cancels queued) | D1 codify-as-live, D2 exemption granted. Manifest `def_hash` == live production hash 10/10 — the ten RPC "divergences" are closed as a mirror gap. §3.3: `token_estimate` was chain-stale (added by `112000`, no-op on prod); `rag_visual_eval_*` id defaults rebound to `pg_catalog.gen_random_uuid()` (no-op on prod); (d) production's restored `document_chunks_content_trgm_idx` (`coalesce(content,'')` form, `8499c3d3…`) IS canonical — staging carries the 2026-06-06 form; repair = Phase 4.4 guard migration + staging drop-and-recreate. Staging at 199 migrations, single drift residual = that index. **New finding:** `20260712000000`'s owner step never landed for three hybrid RPCs — chain-stale, forward-codified by `113000` (not a production hole). Tooling: `scripts/check-drift.ts:192` clips column diffs to 240 chars (never names the column) — queued P3; PR template lacks `RAG impact:` guidance — queued P3. | +| 4 — Index restoration | ☑ COMPLETE 2026-08-19 — production **and** staging | **#2151** (branch `claude/db-remediation-phase4-indexes-a1661a`, head `5d3dca4dd`); forensics §Phase 4 completion; live-drift run `32171070287` | Owner-authorised off-peak window, D4 **OFF** so nothing auto-deployed. All **20** `missing_live` indexes rebuilt with `CREATE INDEX CONCURRENTLY` from canonical definitions — Batch A 14/14, Batch B 6/6, every one `indisvalid AND indisready` with normalised `pg_get_indexdef` matching canonical; zero invalid builds, zero retries, zero skips, zero lock waits; no transactional build attempted; `#102` held out. Both `unexpected_live` orphans **DROPPED** concurrently (not codified) — each is a strict leading-column subset of a present canonical index and the chain already commands both drops (`20260620000000`, `20260703030000`/`20260708000000`); neither is in `schema.sql`, so the drop moves live _into_ mirror agreement. Live now 210 indexes vs manifest 210, zero invalid anywhere. Codified by real `db push` (never `migration repair`; all four history rows carry executed statements): `20260819100000`/`100100` guard Batch A/B, **`20260819100200` discharges the 4.4 trigram-guard debt** that `20260804110240` never covered, `20260819100300` takes `required_indexes` 22→30 adopting **all 8** Phase 6.3 monitor-candidates (`search-health-unmonitored-indexes.json` 44→36, no `monitor-candidate` left; production `ok: true`). **Staging also closed:** `document_chunks_content_trgm_idx` rebuilt into the canonical `coalesce(content,'')` form (§3.3(d)), the four migrations applied by the Phase 2 method with all md5s matching, `check:drift` vs staging **green, 0 unexpected (was 19)**, corpus untouched. Live-drift **37 → 16**: `missing_live` **20 → 0**, `unexpected_live` **2 → 0**; the lone function mismatch is merge-pending and proven (live `def_hash 85df52de…` == this branch's regenerated manifest). **Two escalations, not absorbed:** PITR is OFF on production (new P2 queued) and the `migration_history` block did **not** drop — no allowlist entry was earned (empty intersection), so the 15 stay `#Q5JHBJ`'s work. | +| 5 — Measure | ◐ PARTIAL (incident) | #1960 §Phase 5 | Text fast path 31,610 → 1,535 ms; hybrid 8,519 ms residual. Full EXPLAIN set, `work_mem` before/after (only if D1 changes values), `check:production-readiness` pending. | +| 6 — Future-proofing | ◐ REPO-SIDE COMPLETE 2026-08-18 | **#2058** merged `9c660af1f` (14 paths verified on `main`); follow-up #2090 merged `79e5d2a68` | 6.1 migration `20260818090000_schema_drift_snapshot_history_probe.sql` built + mirrored + manifest regenerated, **LIVE on production since 2026-08-18** (auto-applied on merge; §3.7 verified probe `ok`, 20 `migration_history` rows = 15 unallowlisted §1.1 + 5 seeded). Next for 6.2: fail-fast guard migrations for the 15 — never bare allowlisting. 6.2 contract + `supabase/drift-allowlist.json` (5 seeded superseded guards; ~18 §1.1 versions deliberately NOT allowlisted → first post-deploy run reports them; fix = fail-fast guard migrations, never bare allowlisting). 6.3 `tests/search-health-index-coverage.test.ts` + unmonitored list (44 entries, 8 monitor-candidates). New P2 row queued (inbox `d6ce8a1d`). Stale branch `claude/database-drift-allowlist-48839e` (tip `3fe1deafe`) was recreated after merge — redundant, owner may delete. | +| 7 — Deferred | ☐ not started | — | Each its own session, per playbook. | **Owner decisions (2026-08-18):** D1 = codify-as-live (128 MB on the four hybrids, 64 MB on the other six) — DECIDED. D2 = `work_mem` eval-canary exemption — GRANTED. D3 = 6.1 deploy bundled into -the Phase 3 production window — DECIDED. +the Phase 3 production window — DECIDED. D4 = Supabase GitHub auto-deploy **disabled** — DECIDED (confirmed empirically in the 2026-08-19 Phase 4 window). **Production window — CLOSED 2026-08-18 without a push (PR #2123 → forensics §3.7).** The authorised window's pre-flight `supabase migration list` showed all five `20260818*` migrations @@ -105,13 +105,18 @@ no mark-applied path was used, and production received zero writes. Verified rea `ok`; ten `work_mem` values = D1; live-drift run `32131517648` = **0 function mismatches**, 20 `missing_live`, 2 `unexpected_live`, 15 `migration_history`. The RPC track of `#316` is closed. -**KEY FINDING — D4 pending (owner decision).** The Supabase GitHub integration (Branching, production +**KEY FINDING — D4 DECIDED (auto-deploy disabled).** The Supabase GitHub integration (Branching, production bound to git `main`, branch record from 2026-06-27) auto-applies every migration merged to `main` onto production — live-drift bracketed `110000–112000` to ~34 s after #2106 merged. This undermines the explicit-window model the plan and playbook assume: a merged migration IS a production deploy. -**Recommendation:** disable "Deploy to production" (Supabase dashboard → Project Settings → -Integrations → GitHub) until D4 is decided deliberately; owner-only setting, untouched by any chat. -Either way, Phase 4 is designed to be safe under auto-deploy (prebuild + guard). +**Resolved 2026-08-19:** D4 was decided OFF and "Deploy to production" is disabled (stated in the +Phase 4 task brief; who changed the setting and when is not recorded here). The Phase 4 window confirmed it +empirically — `supabase migration list` showed the four new `20260819*` versions still pending on +production after the branch existed, so nothing had been applied by merge, and they reached production +only via an explicit `supabase db push`. **The explicit-window model in the plan and playbook is therefore +live again and must be honoured:** a merged migration is no longer a production deploy, so every future +phase needs its own approved window and its own push. Phase 4 was built to be safe either way +(prebuild + validate-only guard) and that pattern stays mandatory. **Ledger state note (2026-08-19):** inbox on `main` holds 23 pending requests (remediation: `#248`/`#183`/`#318`/… `done`s from other streams, several `add`s incl. the review-bot budget P1); @@ -123,6 +128,18 @@ open PR #2130 queues 7 more requests but carries **no** reconcile transaction. O link a dedicated worktree for production reads and `supabase unlink` after; `supabase db query --linked --project-ref ` works read-only via the management API without a DB password. +**Where the programme stands after Phase 4 (2026-08-19).** The index track of `#316` is closed on both +tiers and staging is at full parity, so the remaining live-drift findings are exactly one category: +the fifteen `migration_history` no-statements rows, which are `#Q5JHBJ` and need fail-fast guard +migrations, never bare allowlisting. Phase 4 earned no allowlist entry for any of them — measured, not +skipped: six of the fifteen are index-shaped and the intersection between the objects they create and +the twenty-two Phase 4 guards validate is empty. **Next dispatches:** `#Q5JHBJ` (Phase 6.2 guard +migrations for the fifteen) and Phase 5 close-out (after-EXPLAIN set, `#231` re-test on healthy +latency, `check:production-readiness`). **Blocking the owner, not a worker:** PITR is off on production +(new P2 queued from the Phase 4 window), which means the plan's standing "restore point before any +mutating phase" rule cannot be met — no future window that mutates DATA rather than indexes should +proceed on Phase 4's index-only precedent until that is enabled or the rule is deliberately amended. + ## Coordination rules - **One worker chat per phase**, launched with that phase's prompt from the playbook, in the diff --git a/docs/outstanding-issues-inbox/07867a60-274a-4350-a564-2fed480ff997.json b/docs/outstanding-issues-inbox/07867a60-274a-4350-a564-2fed480ff997.json new file mode 100644 index 0000000000..4600908746 --- /dev/null +++ b/docs/outstanding-issues-inbox/07867a60-274a-4350-a564-2fed480ff997.json @@ -0,0 +1,14 @@ +{ + "version": 2, + "id": "07867a60-274a-4350-a564-2fed480ff997", + "createdOn": "2026-08-18", + "action": "add", + "payload": { + "pri": "P2", + "type": "issue", + "summary": "Point-in-time recovery is OFF on the live Supabase project, so the remediation plan's 'restore point before any mutating phase' rule cannot be met", + "detail": "Measured 2026-08-19 during the Phase 4 production window (forensics section 'Phase 4 completion', step 1). `supabase backups list --project-ref sjrfecxgysukkwxsowpy` returns pitr_enabled false, walg_enabled true, and seven retained daily physical backups, most recent COMPLETED 2026-08-17T20:33:28Z -- roughly 38 hours before that window opened. CONSEQUENCE: the recovery-point objective for the live clinical corpus (2851 documents, 70120 chunks) is up to ~24 hours, and the database-remediation plan's standing rule 'PITR/backup restore point captured before any mutating phase' CANNOT be satisfied on this project as currently configured. Phase 4 proceeded anyway and said so: every statement in it was index-only with an exact one-statement inverse (CREATE INDEX CONCURRENTLY <-> DROP INDEX CONCURRENTLY) and no data-loss surface, the same reasoning the 2026-08-14 incident window recorded. That reasoning does NOT generalise. Any future window that mutates DATA rather than indexes -- #022's BMJ attestation hosted apply, #036's public_corpus marker migration, #191's ACL consolidation, #057's restore/soak drill, or any reindex or backfill -- must not proceed on this precedent while the only restore point is a day-old physical backup. DECISION REQUIRED FROM THE OWNER, and it is dashboard plus billing work no agent can do: either enable PITR on the Supabase project (Database settings -> Add-ons -> Point in Time Recovery; it is a paid add-on, so this is a cost decision as well as a safety one), or deliberately accept the ~24h RPO and amend the plan's standing rule to say so, so that future sessions stop treating an unsatisfiable checklist item as if it had been met. Whichever is chosen, record it, because the current state is that the rule reads as satisfied by default when nobody checks. Consider re-grading this row to P1 if a ~24-hour worst-case data loss on the clinical corpus is judged unacceptable; it is filed P2 because the exposure is a deliberate platform configuration rather than a regression, and no data loss has occurred. Related: #057 (restore/rollback drill -- its value is limited while PITR is off), #188 and #196-#200 (DR codification).", + "source": "Phase 4 production window 2026-08-19; docs/audit/live-drift-forensics-2026-08.md 'Phase 4 completion' step 1; PR #2151", + "issueUlid": "01M0B4V6S21K6T35KEB130CXJB" + } +} diff --git a/docs/outstanding-issues-inbox/2c95ad4c-bbc7-4043-9c79-8afd529fa4c2.json b/docs/outstanding-issues-inbox/2c95ad4c-bbc7-4043-9c79-8afd529fa4c2.json new file mode 100644 index 0000000000..8c5d72b944 --- /dev/null +++ b/docs/outstanding-issues-inbox/2c95ad4c-bbc7-4043-9c79-8afd529fa4c2.json @@ -0,0 +1,10 @@ +{ + "version": 2, + "id": "2c95ad4c-bbc7-4043-9c79-8afd529fa4c2", + "createdOn": "2026-08-18", + "action": "cancel", + "payload": { + "requestId": "7a93a5a2-d3b9-4f3d-9747-22049300cd64", + "reason": "Superseded and partly stale. Queued by the 2026-08-18 coordination session before Phase 4 ran; it records D4 as 'owner decision pending' and lists Phase 4 as REMAINING, both of which are now wrong (D4 is decided OFF, Phase 4 completed 2026-08-19). Its evidence is preserved in full in docs/audit/live-drift-forensics-2026-08.md section 3.7 and on the coordination board, and its still-current facts (the RPC-divergence track closing, the D4 platform finding) are carried forward verbatim into the replacement update queued in this same batch. Cancelled together with b048dbdd so exactly one mutation lands on #316." + } +} diff --git a/docs/outstanding-issues-inbox/51673c73-b0ea-4848-bae4-83a269dfa833.json b/docs/outstanding-issues-inbox/51673c73-b0ea-4848-bae4-83a269dfa833.json new file mode 100644 index 0000000000..430b230723 --- /dev/null +++ b/docs/outstanding-issues-inbox/51673c73-b0ea-4848-bae4-83a269dfa833.json @@ -0,0 +1,12 @@ +{ + "version": 2, + "id": "51673c73-b0ea-4848-bae4-83a269dfa833", + "createdOn": "2026-08-18", + "action": "update", + "payload": { + "id": "#316", + "detail": "PHASE 4 COMPLETE 2026-08-19; the RPC track of this row closed 2026-08-18. BOTH HALVES OF THIS ROW ARE NOW CLOSED. (A) RPC divergence, from the 2026-08-18 window (PR #2123, forensics 3.7): the authorised window's pre-flight found the pending set EMPTY -- all five 20260818 migrations were already applied with executed statements (3/11/12/5/4, the CLI db push shape, not mark-applied) -- so db push was never run, no migration repair, no vault reads, zero production writes. Manifest def_hash equals live for all ten match_* functions, and live-drift 32131517648 showed 0 function mismatches. CAUSE, platform finding since resolved: the Supabase GitHub integration (Branching, production bound to git main, branch record 2026-06-27) was auto-applying every migration merged to main -- push-triggered live-drift bracketed 110000-112000 to 34 s after #2106's squash-merge. D4 IS NOW DECIDED: auto-deploy is OFF, confirmed empirically on 2026-08-19 when the four new 20260819 migrations sat pending on production after the branch existed and reached it only via an explicit db push. Not established: who enabled the integration or when, or whether the July mark-applied rows trace to it. (B) Index restoration, 2026-08-19 owner-authorized off-peak window: all 20 missing_live indexes rebuilt with CREATE INDEX CONCURRENTLY from canonical definitions cross-read against their defining migrations -- Batch A 14/14, Batch B 6/6, every one indisvalid AND indisready with normalized pg_get_indexdef matching canonical, zero invalid builds, zero retries, zero skips, zero lock waits (pg_locks read between every Batch B build). No transactional build was ever attempted; #102's bare-column indexes held out. Both unexpected_live indexes were DROPPED CONCURRENTLY rather than codified, because the chain already commands both drops and each is a strict leading-column subset of a present canonical index: document_table_facts_document_id_idx (superseded per 20260620000000) and storage_cleanup_jobs_owner_id_idx (superseded per 20260703030000/20260708000000). Live now reports 210 public indexes against the manifest's 210, zero invalid anywhere. Codified in five migrations applied by real supabase db push (never migration repair; every history row carries executed statements): 20260819100000/100100 guard Batch A/B, 20260819100200 discharges the plan 4.4 debt by guarding the two trigram indexes restored 2026-08-14 that 20260804110240 never checked, 20260819100300 takes search_schema_health() required_indexes 22->30 adopting all 8 Phase 6.3 monitor-candidates (unmonitored list 44->36, no monitor-candidate left; production ok true), and 20260819100150 repairs a chain defect the guard itself caught -- see below. Live-drift 32171070287: UNEXPECTED DRIFT 37->16, missing_live 20->ZERO, unexpected_live 2->ZERO. Staging brought to full parity in the same task; its drift comparison is GREEN with ZERO unexpected drift (was 19), corpus untouched, --prune-stale correctly not used. THE GUARD EARNED ITS KEEP: 20260819100200 failed the Supabase Preview check on PR #2151 because a preview branch builds from the migration chain alone, and the chain permanently produced the WRONG document_chunks_content_trgm_idx -- 20260606000000 creates it first without coalesce(content,''), and both later correct creators use IF NOT EXISTS so they no-op, with no migration ever dropping it. Forensics 3.3(d) had scoped this as staging-only and hand-repaired it there; it was never staging-only (db reset, DR replay, CI migration replay, preview branches all get the wrong index, which is NULL for rows with NULL content and so silently omits those chunks). Fixed by 20260819100150, conditional so it no-ops when canonical, rebuilds only on an empty table, and raises rather than run a write-blocking build on a populated one; proven by replaying the whole chain into a scratch Postgres (fails without it exactly as CI did, 204/204 with it) and the no-op path proven on production itself (index OID unchanged at 1491258 across the push). TWO ESCALATIONS FOR THE OWNER, neither absorbed. (1) PITR IS NOT ENABLED on production (pitr_enabled false, walg_enabled true, daily physical backups only, latest 2026-08-17T20:33:28Z), so the plan's standing 'restore point before any mutating phase' rule cannot be met; Phase 4 proceeded only because every statement was index-only with an exact one-statement inverse, and no future window that mutates DATA should proceed on that precedent. Queued separately as its own P2. (2) The migration_history block did NOT drop and no allowlist entry was written -- measured, not skipped: of the 15 no-statements versions, 6 are index-shaped and the intersection between the objects they create and the 22 these guards validate is EMPTY (near-misses are distinct objects, e.g. audit_logs_owner_id_idx vs audit_logs_owner_created_idx). The 15 stay unallowlisted and remain #Q5JHBJ's work. REMAINING FOR THIS ROW: nothing on the index or RPC tracks. Phase 5 measurement (after-EXPLAIN set, #231 re-test on healthy latency, check:production-readiness) is the only follow-on. Full evidence with dates, run IDs and pasted output in docs/audit/live-drift-forensics-2026-08.md sections 3.7 and 'Phase 4 completion'. Session traps still current: the main checkout D:\\Repos\\Database is linked to STAGING, so link a dedicated worktree for production and unlink after; supabase db query --linked --project-ref works read-only via the management API without a DB password; db query parses a leading -- as a flag, so pass SQL that starts with a comment via --file; production has no track_commit_timestamp.", + "source": "PR #2151 (Phase 4) and PR #2123 (window 3.7); live-drift runs 32171070287 and 32131517648; forensics sections 3.7 and 'Phase 4 completion'", + "baseRowFingerprint": "de004897b857158a00169721415caf512defac916bbff0466da5d7d3da4049c4" + } +} diff --git a/docs/outstanding-issues-inbox/946d0cce-549d-4caf-b74d-af5593e0911b.json b/docs/outstanding-issues-inbox/946d0cce-549d-4caf-b74d-af5593e0911b.json new file mode 100644 index 0000000000..9abe6ed5e6 --- /dev/null +++ b/docs/outstanding-issues-inbox/946d0cce-549d-4caf-b74d-af5593e0911b.json @@ -0,0 +1,10 @@ +{ + "version": 2, + "id": "946d0cce-549d-4caf-b74d-af5593e0911b", + "createdOn": "2026-08-18", + "action": "cancel", + "payload": { + "requestId": "b048dbdd-7672-4e47-82e2-2211495e40c6", + "reason": "Cancelled by its own author to resolve a two-pending-mutations collision on #316. Its content is carried forward unchanged into the merged replacement queued in this same batch, which additionally carries the RPC-track closure and D4 finding from the concurrently queued 7a93a5a2 so neither session's result is lost." + } +} diff --git a/docs/outstanding-issues-inbox/b048dbdd-7672-4e47-82e2-2211495e40c6.json b/docs/outstanding-issues-inbox/b048dbdd-7672-4e47-82e2-2211495e40c6.json new file mode 100644 index 0000000000..549f8e6b96 --- /dev/null +++ b/docs/outstanding-issues-inbox/b048dbdd-7672-4e47-82e2-2211495e40c6.json @@ -0,0 +1,11 @@ +{ + "version": 2, + "id": "b048dbdd-7672-4e47-82e2-2211495e40c6", + "createdOn": "2026-08-18", + "action": "update", + "payload": { + "id": "#316", + "detail": "PHASE 4 COMPLETE 2026-08-19 in an owner-authorized off-peak production window (D4 auto-deploy OFF, so nothing reached production on merge; every hosted change was made by an explicit named step). The index track of this row is CLOSED. All 20 missing_live indexes were rebuilt with CREATE INDEX CONCURRENTLY using canonical definitions cross-read against their defining migrations: Batch A 14/14 and Batch B 6/6, every one indisvalid AND indisready with normalized pg_get_indexdef matching canonical, zero invalid builds, zero retries, zero skips, zero lock waits (pg_locks read between every Batch B build, waiting stayed 0). No transactional build was ever attempted and #102's bare-column indexes were held out entirely. Both unexpected_live indexes were DROPPED CONCURRENTLY rather than codified, because the repo chain already commands both drops and each is a strict leading-column subset of a present canonical index: document_table_facts_document_id_idx (superseded by document_table_facts_document_idx per 20260620000000) and storage_cleanup_jobs_owner_id_idx (superseded by storage_cleanup_jobs_owner_status_idx per 20260703030000/20260708000000). Live now reports 210 public indexes against the manifest's 210 repo-defined, with zero invalid-or-not-ready anywhere. Codified in four migrations pushed to production by real supabase db push (never migration repair --status applied; all four history rows carry executed statements, stmt_count 4, so none can ever appear in the migration_history probe): 20260819100000 and 20260819100100 guard Batch A and Batch B, 20260819100200 discharges the plan 4.4 debt by guarding the two trigram indexes restored 2026-08-14 that 20260804110240 never checked, and 20260819100300 extends search_schema_health() required_indexes from 22 to 30. All three guards were dry-run green against production BEFORE the push. required_indexes decision: all 8 Phase 6.3 monitor-candidates are now monitored (the 3 formerly-absent ones only after their builds were confirmed), leaving search-health-unmonitored-indexes.json at 36 entries with no monitor-candidate remaining; production search_schema_health() reports ok true, missing []. Live-drift dispatch on main, Actions run 32171070287: UNEXPECTED DRIFT fell from 37 to 16 — missing_live 20 to ZERO, unexpected_live 2 to ZERO. The single remaining function mismatch is search_schema_health() and is merge-pending, proven not assumed: live's def_hash 85df52de66e4e89d4a328b81a3a87c90 is byte-identical to this branch's regenerated manifest, so it clears when the PR lands. Staging (ikoiolksxqxfxgiyqpnu) was brought to full parity in the same task: document_chunks_content_trgm_idx was dropped and rebuilt into the canonical coalesce(content,'') form per forensics 3.3(d), the four migrations were applied by the Phase 2 method with all four md5s matching their repo files, and the drift comparison against staging is now GREEN with ZERO unexpected drift (was 19), corpus untouched at 0 documents and --prune-stale correctly not used. TWO ESCALATIONS FOR THE OWNER, neither absorbed. (1) PITR IS NOT ENABLED on production: backups list reports pitr_enabled false, walg_enabled true, and only daily physical backups, latest 2026-08-17T20:33:28Z. Step 1 of the authorized procedure could not be satisfied. This window proceeded because every statement was index-only with an exact one-statement inverse and no data-loss surface, but the plan's standing 'restore point before any mutating phase' rule cannot be met on this project as configured, and no future phase that mutates DATA should proceed on this precedent. (2) The migration_history block did NOT drop and no allowlist entry was written, which is a measured result: of the 15 no-statements versions, 6 are index-shaped, and the intersection between the objects they create and the 22 objects these guards validate is EMPTY (near-misses are distinct objects, e.g. audit_logs_owner_id_idx vs audit_logs_owner_created_idx). No honest validation-class entry existed, so the 15 stay unallowlisted and remain #Q5JHBJ's work. Full evidence with dates, run IDs and pasted output in docs/audit/live-drift-forensics-2026-08.md section Phase 4 completion.", + "baseRowFingerprint": "de004897b857158a00169721415caf512defac916bbff0466da5d7d3da4049c4" + } +} diff --git a/supabase/drift-manifest.json b/supabase/drift-manifest.json index e914ca130b..1bc9e2e45d 100644 --- a/supabase/drift-manifest.json +++ b/supabase/drift-manifest.json @@ -1,9 +1,9 @@ { - "generated_at": "2026-08-18T08:30:22.062Z", + "generated_at": "2026-08-18T18:15:50.121Z", "generator": "scripts/generate-drift-manifest.ts", "postgres_image": "supabase/postgres:17.6.1.127@sha256:be60aee15997daca475b710b734bc6bfe52cd544dcd7e9fd2ff58210b6747d83", - "schema_sha256": "87ac9fc4849ee9cf0404a7260ef1b8bf08a14067573f3a87574a14bbc10469cf", - "replay_seconds": 75, + "schema_sha256": "328677d1c6f3e0136ca108e8117a73051553a2387b078836001e8cdc96f4c2d3", + "replay_seconds": 21, "snapshot": { "views": [ { @@ -7336,7 +7336,7 @@ "postgres=X/postgres", "service_role=X/postgres" ], - "def_hash": "f4f5f536026c4dd27d506a8e40b8c6d7", + "def_hash": "85df52de66e4e89d4a328b81a3a87c90", "signature": "public.search_schema_health()" }, { diff --git a/supabase/migrations/20260819100000_restore_batch_a_operational_indexes.sql b/supabase/migrations/20260819100000_restore_batch_a_operational_indexes.sql new file mode 100644 index 0000000000..876f864581 --- /dev/null +++ b/supabase/migrations/20260819100000_restore_batch_a_operational_indexes.sql @@ -0,0 +1,152 @@ +-- Phase 4 Batch A (plan section 4.1): record the already-completed restoration of the +-- fourteen repo-defined operational indexes that live-drift reported as missing_live +-- (forensics section 1.3, 2026-08-14; still absent at the 2026-08-19 window pre-flight). +-- +-- Every definition below is the canonical one already carried by supabase/schema.sql and by +-- the migration that first created it, so this migration adds no schema and schema.sql needs +-- no mirror change. Like 20260804110240 it VALIDATES and never builds: a plain CREATE INDEX +-- here would hold write-blocking locks, and CREATE INDEX CONCURRENTLY cannot run inside the +-- per-migration transaction that supabase db push uses. A drifted hosted target must prebuild +-- each index with CREATE INDEX CONCURRENTLY outside any transaction, validate pg_index +-- indisvalid/indisready plus the canonical definition, and only then apply this version. +-- This guard fails fast if that operator step was skipped. +-- +-- Timeouts use SET LOCAL so they do not leak into later migrations applied on the same CLI +-- session connection (plain SET is session-scoped). +-- +-- Name resolution is canonical-only: none of these fourteen has an entry in +-- search_schema_health()'s index_aliases map. If an alias is ever added for one of them, +-- update this guard before relying on the health probe as evidence of repair. + +set local search_path = public, extensions, pg_catalog; +set local lock_timeout = '5s'; +set local statement_timeout = '30s'; + +do $migration$ +declare + missing_indexes text[] := array[]::text[]; + invalid_indexes text[] := array[]::text[]; + mismatched_indexes text[] := array[]::text[]; + required record; + index_oid regclass; + is_valid boolean; + is_ready boolean; + actual_def text; + actual_normalized text; + expected_normalized text; +begin + for required in + select * + from ( + values + ( + 'audit_logs_action_created_idx', + 'create index audit_logs_action_created_idx on public.audit_logs using btree (action, created_at desc)' + ), + ( + 'audit_logs_owner_created_idx', + 'create index audit_logs_owner_created_idx on public.audit_logs using btree (owner_id, created_at desc)' + ), + ( + 'api_rate_limits_bucket_updated_idx', + 'create index api_rate_limits_bucket_updated_idx on public.api_rate_limits using btree (bucket, updated_at desc)' + ), + ( + 'rag_aliases_type_enabled_idx', + 'create index rag_aliases_type_enabled_idx on public.rag_aliases using btree (alias_type, enabled)' + ), + ( + 'rag_queries_source_chunk_ids_gin_idx', + 'create index rag_queries_source_chunk_ids_gin_idx on public.rag_queries using gin (source_chunk_ids)' + ), + ( + 'rag_query_misses_aliases_idx', + 'create index rag_query_misses_aliases_idx on public.rag_query_misses using gin (candidate_aliases)' + ), + ( + 'image_caption_cache_owner_hash_idx', + 'create index image_caption_cache_owner_hash_idx on public.image_caption_cache using btree (owner_id, image_hash, model)' + ), + ( + 'document_index_quality_owner_score_idx', + 'create index document_index_quality_owner_score_idx on public.document_index_quality using btree (owner_id, quality_score, updated_at desc)' + ), + ( + 'document_publication_approvals_document_idx', + 'create index document_publication_approvals_document_idx on public.document_publication_approvals using btree (document_id, approved_at desc)' + ), + ( + 'document_summaries_owner_idx', + 'create index document_summaries_owner_idx on public.document_summaries using btree (owner_id, generated_at desc)' + ), + ( + 'indexing_v3_agent_jobs_locked_at_idx', + 'create index indexing_v3_agent_jobs_locked_at_idx on public.indexing_v3_agent_jobs using btree (locked_at) where (status = ''processing''::text)' + ), + ( + 'ingestion_job_stages_job_stage_started_idx', + 'create index ingestion_job_stages_job_stage_started_idx on public.ingestion_job_stages using btree (job_id, stage_name, started_at desc)' + ), + ( + 'medication_records_owner_category_idx', + 'create index medication_records_owner_category_idx on public.medication_records using btree (owner_id, category)' + ), + ( + 'storage_cleanup_jobs_owner_status_idx', + 'create index storage_cleanup_jobs_owner_status_idx on public.storage_cleanup_jobs using btree (owner_id, status, created_at desc)' + ) + ) as t(index_name, canonical_def) + loop + index_oid := to_regclass(format('public.%I', required.index_name)); + if index_oid is null then + missing_indexes := array_append(missing_indexes, required.index_name); + continue; + end if; + + select i.indisvalid, i.indisready, pg_get_indexdef(i.indexrelid) + into is_valid, is_ready, actual_def + from pg_index as i + where i.indexrelid = index_oid; + + if not coalesce(is_valid, false) or not coalesce(is_ready, false) then + invalid_indexes := array_append(invalid_indexes, required.index_name); + continue; + end if; + + -- Keep in lockstep with normalizeIndexDefinition in tests/supabase-schema.test.ts. + actual_normalized := lower(actual_def); + actual_normalized := replace(actual_normalized, 'create index if not exists', 'create index'); + actual_normalized := regexp_replace(actual_normalized, ' extensions\.', ' ', 'g'); + actual_normalized := regexp_replace(actual_normalized, ' using btree', '', 'g'); + actual_normalized := regexp_replace(actual_normalized, 'where \(([^()]*)\)$', 'where \1'); + actual_normalized := replace(actual_normalized, ';', ''); + actual_normalized := regexp_replace(actual_normalized, '[[:space:]]+', ' ', 'g'); + actual_normalized := regexp_replace(actual_normalized, ' on ([^ ()]+) \(', ' on \1(', 'g'); + actual_normalized := btrim(actual_normalized); + + expected_normalized := lower(required.canonical_def); + expected_normalized := replace(expected_normalized, 'create index if not exists', 'create index'); + expected_normalized := regexp_replace(expected_normalized, ' extensions\.', ' ', 'g'); + expected_normalized := regexp_replace(expected_normalized, ' using btree', '', 'g'); + expected_normalized := regexp_replace(expected_normalized, 'where \(([^()]*)\)$', 'where \1'); + expected_normalized := replace(expected_normalized, ';', ''); + expected_normalized := regexp_replace(expected_normalized, '[[:space:]]+', ' ', 'g'); + expected_normalized := regexp_replace(expected_normalized, ' on ([^ ()]+) \(', ' on \1(', 'g'); + expected_normalized := btrim(expected_normalized); + + if actual_normalized is distinct from expected_normalized then + mismatched_indexes := array_append(mismatched_indexes, required.index_name); + end if; + end loop; + + if cardinality(missing_indexes) > 0 + or cardinality(invalid_indexes) > 0 + or cardinality(mismatched_indexes) > 0 then + raise exception + 'Phase 4 Batch A operational indexes were not prebuilt; create the missing indexes concurrently outside the migration transaction, validate them, then apply this version. Missing: %; Invalid: %; Mismatched: %', + coalesce(nullif(array_to_string(missing_indexes, ', '), ''), '(none)'), + coalesce(nullif(array_to_string(invalid_indexes, ', '), ''), '(none)'), + coalesce(nullif(array_to_string(mismatched_indexes, ', '), ''), '(none)'); + end if; +end +$migration$; diff --git a/supabase/migrations/20260819100100_restore_batch_b_retrieval_indexes.sql b/supabase/migrations/20260819100100_restore_batch_b_retrieval_indexes.sql new file mode 100644 index 0000000000..d071a379b0 --- /dev/null +++ b/supabase/migrations/20260819100100_restore_batch_b_retrieval_indexes.sql @@ -0,0 +1,119 @@ +-- Phase 4 Batch B (plan section 4.2): record the already-completed restoration of the six +-- repo-defined indexes on the large retrieval tables that live-drift reported as missing_live +-- (forensics section 1.3, 2026-08-14; still absent at the 2026-08-19 window pre-flight). +-- +-- These sit on documents, document_images, document_chunks and document_index_units, which +-- carry real volume on production, so the concurrent-prebuild requirement is not optional +-- here: a transactional build would block writes on the retrieval path for its duration. +-- Like 20260804110240 this migration VALIDATES and never builds. Definitions are the +-- canonical ones already in supabase/schema.sql, so no mirror change accompanies it. +-- +-- Timeouts use SET LOCAL so they do not leak into later migrations applied on the same CLI +-- session connection (plain SET is session-scoped). +-- +-- Name resolution is canonical-only: none of these six has an entry in +-- search_schema_health()'s index_aliases map. Three of them (document_chunks_anchor_idx, +-- document_index_units_heading_path_idx, documents_registry_projection_lookup_idx) enter +-- required_indexes in 20260819100300, which is what makes their next disappearance visible +-- to the runtime probe rather than only to the weekly drift check. + +set local search_path = public, extensions, pg_catalog; +set local lock_timeout = '5s'; +set local statement_timeout = '30s'; + +do $migration$ +declare + missing_indexes text[] := array[]::text[]; + invalid_indexes text[] := array[]::text[]; + mismatched_indexes text[] := array[]::text[]; + required record; + index_oid regclass; + is_valid boolean; + is_ready boolean; + actual_def text; + actual_normalized text; + expected_normalized text; +begin + for required in + select * + from ( + values + ( + 'documents_registry_projection_lookup_idx', + 'create index documents_registry_projection_lookup_idx on public.documents using btree (((metadata ->> ''registry_record_kind''::text)), ((metadata ->> ''registry_record_id''::text))) where ((metadata ->> ''source_kind''::text) = ''registry_record''::text)' + ), + ( + 'document_images_hash_idx', + 'create index document_images_hash_idx on public.document_images using btree (document_id, image_hash) where (image_hash is not null)' + ), + ( + 'document_images_structured_profile_gin_idx', + 'create index document_images_structured_profile_gin_idx on public.document_images using gin (((metadata -> ''structured_visual_profile''::text)))' + ), + ( + 'document_images_visual_intelligence_version_idx', + 'create index document_images_visual_intelligence_version_idx on public.document_images using btree (((metadata ->> ''visual_intelligence_version''::text))) where (metadata ? ''visual_intelligence_version''::text)' + ), + ( + 'document_chunks_anchor_idx', + 'create index document_chunks_anchor_idx on public.document_chunks using btree (document_id, anchor_id) where (anchor_id is not null)' + ), + ( + 'document_index_units_heading_path_idx', + 'create index document_index_units_heading_path_idx on public.document_index_units using gin (heading_path)' + ) + ) as t(index_name, canonical_def) + loop + index_oid := to_regclass(format('public.%I', required.index_name)); + if index_oid is null then + missing_indexes := array_append(missing_indexes, required.index_name); + continue; + end if; + + select i.indisvalid, i.indisready, pg_get_indexdef(i.indexrelid) + into is_valid, is_ready, actual_def + from pg_index as i + where i.indexrelid = index_oid; + + if not coalesce(is_valid, false) or not coalesce(is_ready, false) then + invalid_indexes := array_append(invalid_indexes, required.index_name); + continue; + end if; + + -- Keep in lockstep with normalizeIndexDefinition in tests/supabase-schema.test.ts. + actual_normalized := lower(actual_def); + actual_normalized := replace(actual_normalized, 'create index if not exists', 'create index'); + actual_normalized := regexp_replace(actual_normalized, ' extensions\.', ' ', 'g'); + actual_normalized := regexp_replace(actual_normalized, ' using btree', '', 'g'); + actual_normalized := regexp_replace(actual_normalized, 'where \(([^()]*)\)$', 'where \1'); + actual_normalized := replace(actual_normalized, ';', ''); + actual_normalized := regexp_replace(actual_normalized, '[[:space:]]+', ' ', 'g'); + actual_normalized := regexp_replace(actual_normalized, ' on ([^ ()]+) \(', ' on \1(', 'g'); + actual_normalized := btrim(actual_normalized); + + expected_normalized := lower(required.canonical_def); + expected_normalized := replace(expected_normalized, 'create index if not exists', 'create index'); + expected_normalized := regexp_replace(expected_normalized, ' extensions\.', ' ', 'g'); + expected_normalized := regexp_replace(expected_normalized, ' using btree', '', 'g'); + expected_normalized := regexp_replace(expected_normalized, 'where \(([^()]*)\)$', 'where \1'); + expected_normalized := replace(expected_normalized, ';', ''); + expected_normalized := regexp_replace(expected_normalized, '[[:space:]]+', ' ', 'g'); + expected_normalized := regexp_replace(expected_normalized, ' on ([^ ()]+) \(', ' on \1(', 'g'); + expected_normalized := btrim(expected_normalized); + + if actual_normalized is distinct from expected_normalized then + mismatched_indexes := array_append(mismatched_indexes, required.index_name); + end if; + end loop; + + if cardinality(missing_indexes) > 0 + or cardinality(invalid_indexes) > 0 + or cardinality(mismatched_indexes) > 0 then + raise exception + 'Phase 4 Batch B retrieval-table indexes were not prebuilt; create the missing indexes concurrently outside the migration transaction, validate them, then apply this version. Missing: %; Invalid: %; Mismatched: %', + coalesce(nullif(array_to_string(missing_indexes, ', '), ''), '(none)'), + coalesce(nullif(array_to_string(invalid_indexes, ', '), ''), '(none)'), + coalesce(nullif(array_to_string(mismatched_indexes, ', '), ''), '(none)'); + end if; +end +$migration$; diff --git a/supabase/migrations/20260819100150_reconcile_chain_stale_content_trgm_index.sql b/supabase/migrations/20260819100150_reconcile_chain_stale_content_trgm_index.sql new file mode 100644 index 0000000000..0d86db0ca8 --- /dev/null +++ b/supabase/migrations/20260819100150_reconcile_chain_stale_content_trgm_index.sql @@ -0,0 +1,118 @@ +-- Chain-stale repair: make the migration chain produce the CANONICAL +-- document_chunks_content_trgm_idx, the one schema.sql, the drift manifest and +-- production already carry. +-- +-- Plan of record: docs/database-remediation-plan.md section 4.4; evidence: +-- docs/audit/live-drift-forensics-2026-08.md section 3.3(d) and section +-- "Phase 4 completion". Ledger anchor #316. +-- +-- THE DEFECT. Three renderings of this index exist in the repository, and the +-- chain picks the wrong one because the first creator wins: +-- +-- 20260606000000:11 lower(coalesce(section_heading,'') || ' ' || content) +-- <- created FIRST, and this is what a replay ends up with +-- 20260622000000:13 lower(coalesce(section_heading,'') || ' ' || coalesce(content,'')) +-- 20260705180000:11 identical to 20260622000000 = schema.sql:743 = CANONICAL +-- +-- Both later creators use CREATE INDEX IF NOT EXISTS, so they are no-ops once +-- 20260606000000 has run, and no migration anywhere drops the index. Any +-- database built from migrations alone therefore gets the 2026-06-06 form, +-- while schema.sql, supabase/drift-manifest.json and production carry the +-- coalesce(content,'') form. The difference is not cosmetic: the 2026-06-06 +-- expression is NULL for every row with a NULL content, so those chunks are +-- absent from the trigram index entirely. +-- +-- This was recorded as a staging-only residual in forensics 3.3(d) and repaired +-- there by hand. It is not staging-only: it is every environment built from the +-- chain -- `supabase db reset`, a disaster-recovery replay, and the Supabase +-- preview branch, where 20260819100200's guard caught it (PR #2151). +-- +-- WHY A CONDITIONAL REPAIR RATHER THAN AN UNCONDITIONAL REBUILD. Production and +-- staging already hold the canonical form, so an unconditional DROP + CREATE +-- would rebuild a 68 MB GIN index over 70k rows inside the migration +-- transaction and block writes on the retrieval path for its duration -- the +-- exact thing this programme forbids. So: +-- +-- * already canonical -> no-op (production, staging today) +-- * wrong form, table EMPTY -> drop and recreate (preview, db reset, DR +-- replay: instant, nothing to block) +-- * wrong form, table POPULATED -> raise, and tell the operator to rebuild +-- concurrently out of band first, per the +-- 20260804110240 pattern +-- +-- The last branch is deliberate: this migration will never perform a +-- write-blocking index build on a populated hosted database. It fails loudly +-- and hands the work to an approved window instead. +-- +-- Ordered at 100150 -- after the Batch B guard (100100) and before the trigram +-- guard (100200) -- so a fresh replay is canonical by the time 20260819100200 +-- validates it. + +set local search_path = public, extensions, pg_catalog; +set local lock_timeout = '5s'; +set local statement_timeout = '60s'; + +do $migration$ +declare + canonical_def constant text := + 'create index document_chunks_content_trgm_idx on public.document_chunks using gin (lower(((coalesce(section_heading, ''''::text) || '' ''::text) || coalesce(content, ''''::text))) extensions.gin_trgm_ops)'; + index_oid regclass; + actual_def text; + actual_normalized text; + expected_normalized text; + row_estimate bigint; +begin + -- Keep in lockstep with normalizeIndexDefinition in tests/supabase-schema.test.ts. + expected_normalized := lower(canonical_def); + expected_normalized := replace(expected_normalized, 'create index if not exists', 'create index'); + expected_normalized := regexp_replace(expected_normalized, ' extensions\.', ' ', 'g'); + expected_normalized := regexp_replace(expected_normalized, ' using btree', '', 'g'); + expected_normalized := regexp_replace(expected_normalized, 'where \(([^()]*)\)$', 'where \1'); + expected_normalized := replace(expected_normalized, ';', ''); + expected_normalized := regexp_replace(expected_normalized, '[[:space:]]+', ' ', 'g'); + expected_normalized := regexp_replace(expected_normalized, ' on ([^ ()]+) \(', ' on \1(', 'g'); + expected_normalized := btrim(expected_normalized); + + index_oid := to_regclass('public.document_chunks_content_trgm_idx'); + + if index_oid is not null then + select pg_get_indexdef(i.indexrelid) + into actual_def + from pg_index as i + where i.indexrelid = index_oid; + + actual_normalized := lower(actual_def); + actual_normalized := replace(actual_normalized, 'create index if not exists', 'create index'); + actual_normalized := regexp_replace(actual_normalized, ' extensions\.', ' ', 'g'); + actual_normalized := regexp_replace(actual_normalized, ' using btree', '', 'g'); + actual_normalized := regexp_replace(actual_normalized, 'where \(([^()]*)\)$', 'where \1'); + actual_normalized := replace(actual_normalized, ';', ''); + actual_normalized := regexp_replace(actual_normalized, '[[:space:]]+', ' ', 'g'); + actual_normalized := regexp_replace(actual_normalized, ' on ([^ ()]+) \(', ' on \1(', 'g'); + actual_normalized := btrim(actual_normalized); + + if actual_normalized = expected_normalized then + -- Production and staging land here: nothing to do, no lock taken. + return; + end if; + end if; + + -- Wrong form (or absent). Only rebuild in-transaction when there is nothing + -- to block; a populated table must be repaired concurrently out of band. + select coalesce(c.reltuples, 0)::bigint + into row_estimate + from pg_class as c + where c.oid = to_regclass('public.document_chunks'); + + if coalesce(row_estimate, 0) > 0 or exists (select 1 from public.document_chunks limit 1) then + raise exception + 'document_chunks_content_trgm_idx is not in canonical form on a populated database (found: %). Rebuild it out of band with DROP INDEX CONCURRENTLY + CREATE INDEX CONCURRENTLY using the 20260705180000 definition, validate indisvalid/indisready, then apply this version. This migration will not run a write-blocking index build on a populated table.', + coalesce(actual_def, '(absent)'); + end if; + + drop index if exists public.document_chunks_content_trgm_idx; + create index document_chunks_content_trgm_idx + on public.document_chunks + using gin (lower(coalesce(section_heading, '') || ' ' || coalesce(content, '')) gin_trgm_ops); +end +$migration$; diff --git a/supabase/migrations/20260819100200_restore_search_health_trigram_indexes.sql b/supabase/migrations/20260819100200_restore_search_health_trigram_indexes.sql new file mode 100644 index 0000000000..c2cfbeaef4 --- /dev/null +++ b/supabase/migrations/20260819100200_restore_search_health_trigram_indexes.sql @@ -0,0 +1,108 @@ +-- Plan section 4.4 debt: the fail-fast guard for the two retrieval-critical trigram indexes +-- restored in the 2026-08-14 incident window (forensics section Phase 4). +-- +-- 20260804110240_restore_rag_search_health_indexes.sql names four OTHER indexes and never +-- checks this pair, so its application gave no existence bound for either of them +-- (forensics section 1.1). That is how both could vanish from production between 2026-07-05 +-- and 2026-08-02 while every migration in the chain still replayed green. This guard closes +-- that hole: a later replay cannot silently proceed if either index disappears again. It +-- VALIDATES and never builds, per the 20260804110240 pattern. +-- +-- Definitions are the canonical 20260705180000 / schema.sql forms - the coalesce(content, '') +-- rendering production has carried since the 2026-08-14 rebuild (forensics section 3.3(d)), +-- not the 2026-06-06 form staging still carries. +-- +-- Timeouts use SET LOCAL so they do not leak into later migrations applied on the same CLI +-- session connection (plain SET is session-scoped). +-- +-- ALIASES: unlike the four indexes in 20260804110240, both of these DO have +-- search_schema_health() index_aliases entries (documents_title_search_tsv_idx / +-- documents_title_search_idx, and document_chunks_search_tsv_idx / +-- document_chunks_search_idx). This guard deliberately resolves canonical names only: an +-- alias satisfying the health probe is not evidence that the canonical trigram index exists, +-- which is precisely the failure this guard exists to catch. + +set local search_path = public, extensions, pg_catalog; +set local lock_timeout = '5s'; +set local statement_timeout = '30s'; + +do $migration$ +declare + missing_indexes text[] := array[]::text[]; + invalid_indexes text[] := array[]::text[]; + mismatched_indexes text[] := array[]::text[]; + required record; + index_oid regclass; + is_valid boolean; + is_ready boolean; + actual_def text; + actual_normalized text; + expected_normalized text; +begin + for required in + select * + from ( + values + ( + 'documents_title_trgm_idx', + 'create index documents_title_trgm_idx on public.documents using gin (lower(((coalesce(title, ''''::text) || '' ''::text) || coalesce(file_name, ''''::text))) extensions.gin_trgm_ops)' + ), + ( + 'document_chunks_content_trgm_idx', + 'create index document_chunks_content_trgm_idx on public.document_chunks using gin (lower(((coalesce(section_heading, ''''::text) || '' ''::text) || coalesce(content, ''''::text))) extensions.gin_trgm_ops)' + ) + ) as t(index_name, canonical_def) + loop + index_oid := to_regclass(format('public.%I', required.index_name)); + if index_oid is null then + missing_indexes := array_append(missing_indexes, required.index_name); + continue; + end if; + + select i.indisvalid, i.indisready, pg_get_indexdef(i.indexrelid) + into is_valid, is_ready, actual_def + from pg_index as i + where i.indexrelid = index_oid; + + if not coalesce(is_valid, false) or not coalesce(is_ready, false) then + invalid_indexes := array_append(invalid_indexes, required.index_name); + continue; + end if; + + -- Keep in lockstep with normalizeIndexDefinition in tests/supabase-schema.test.ts. + actual_normalized := lower(actual_def); + actual_normalized := replace(actual_normalized, 'create index if not exists', 'create index'); + actual_normalized := regexp_replace(actual_normalized, ' extensions\.', ' ', 'g'); + actual_normalized := regexp_replace(actual_normalized, ' using btree', '', 'g'); + actual_normalized := regexp_replace(actual_normalized, 'where \(([^()]*)\)$', 'where \1'); + actual_normalized := replace(actual_normalized, ';', ''); + actual_normalized := regexp_replace(actual_normalized, '[[:space:]]+', ' ', 'g'); + actual_normalized := regexp_replace(actual_normalized, ' on ([^ ()]+) \(', ' on \1(', 'g'); + actual_normalized := btrim(actual_normalized); + + expected_normalized := lower(required.canonical_def); + expected_normalized := replace(expected_normalized, 'create index if not exists', 'create index'); + expected_normalized := regexp_replace(expected_normalized, ' extensions\.', ' ', 'g'); + expected_normalized := regexp_replace(expected_normalized, ' using btree', '', 'g'); + expected_normalized := regexp_replace(expected_normalized, 'where \(([^()]*)\)$', 'where \1'); + expected_normalized := replace(expected_normalized, ';', ''); + expected_normalized := regexp_replace(expected_normalized, '[[:space:]]+', ' ', 'g'); + expected_normalized := regexp_replace(expected_normalized, ' on ([^ ()]+) \(', ' on \1(', 'g'); + expected_normalized := btrim(expected_normalized); + + if actual_normalized is distinct from expected_normalized then + mismatched_indexes := array_append(mismatched_indexes, required.index_name); + end if; + end loop; + + if cardinality(missing_indexes) > 0 + or cardinality(invalid_indexes) > 0 + or cardinality(mismatched_indexes) > 0 then + raise exception + 'The retrieval-critical trigram indexes restored on 2026-08-14 are not present in canonical form; rebuild them concurrently outside the migration transaction, validate them, then apply this version. Missing: %; Invalid: %; Mismatched: %', + coalesce(nullif(array_to_string(missing_indexes, ', '), ''), '(none)'), + coalesce(nullif(array_to_string(invalid_indexes, ', '), ''), '(none)'), + coalesce(nullif(array_to_string(mismatched_indexes, ', '), ''), '(none)'); + end if; +end +$migration$; diff --git a/supabase/migrations/20260819100300_monitor_restored_retrieval_indexes.sql b/supabase/migrations/20260819100300_monitor_restored_retrieval_indexes.sql new file mode 100644 index 0000000000..c5bdd94bb7 --- /dev/null +++ b/supabase/migrations/20260819100300_monitor_restored_retrieval_indexes.sql @@ -0,0 +1,231 @@ +-- Phase 4.4 monitoring ratchet: put the eight Phase 6.3 monitor-candidates into +-- search_schema_health()'s required_indexes list. +-- +-- Plan of record: docs/database-remediation-plan.md sections 4.4 and 6.3; evidence: +-- docs/audit/live-drift-forensics-2026-08.md sections 1.3, Phase 4 and Phase 6. +-- Ledger anchor #316. +-- +-- WHY: search_schema_health() monitors a curated list, and every one of the twenty +-- indexes that went missing on production was invisible to it. The runtime probe +-- reported ok: true for weeks while the retrieval path did sequential scans. The +-- 6.3 ratchet (tests/search-health-index-coverage.test.ts + +-- supabase/search-health-unmonitored-indexes.json) made every monitoring decision on +-- the six retrieval-critical tables explicit and left exactly eight indexes flagged +-- 'monitor-candidate' — deliberately deferred to this migration, because +-- required_indexes changes travel by migration only, never by editing the mirror. +-- +-- All eight are now monitored: +-- * documents_registry_projection_lookup_idx, document_chunks_anchor_idx and +-- document_index_units_heading_path_idx were three of the twenty absent indexes. +-- They were rebuilt concurrently and validated in the 2026-08-19 window +-- (guards 20260819100000 / 20260819100100) BEFORE this migration adds them, so +-- it cannot turn the probe red on a still-absent object. +-- * documents_search_idx, document_embedding_fields_search_tsv_chunk_gin_idx, +-- document_index_units_search_idx, document_index_units_terms_idx and +-- document_memory_cards_search_idx are GIN indexes on the lexical half of the +-- retrieval RPCs, all present and valid on production, none of which had any +-- monitored equivalent. document_index_units was the worst-covered table in the +-- scope at 2 of 16 monitored. +-- +-- After this, supabase/search-health-unmonitored-indexes.json carries no +-- 'monitor-candidate' entries: every remaining entry is a reasoned +-- 'accepted-unmonitored'. The coverage test rejects an entry that is also monitored, +-- so the eight are removed from that file in the same change. +-- +-- SCOPE: this changes which index absences the probe REPORTS. It adds no index, +-- drops none, and touches no retrieval SQL, ranking input or RPC body. The probe +-- feeds /api/setup-status only (src/app/api/setup-status/route.ts) and is not on the +-- answer path, so an expanded list cannot alter retrieval behaviour. +-- +-- The body below is 20260706010000_search_schema_health_m13_guard.sql's definition +-- verbatim, with the eight names appended to required_indexes and nothing else +-- changed; index_aliases, the M13 commit_document_index_generation probe, the +-- hybrid-RPC smoke checks and the grants are all carried over unmodified. + +set search_path = public, extensions, pg_catalog; + +create or replace function public.search_schema_health() +returns jsonb +language plpgsql +stable +security definer +set search_path = public, extensions, pg_catalog, pg_temp +as $$ +declare + missing text[] := array[]::text[]; + vector_type_oid oid; + vector_schema text; + index_name text; + legacy_ivfflat_indexes text[]; + zero_vec extensions.vector(1536); + probe_text text := 'schema health probe zzznomatch'; + hybrid_rpcs text[] := array[ + 'match_document_chunks_hybrid', + 'match_document_index_units_hybrid', + 'match_document_embedding_fields_hybrid', + 'match_document_memory_cards_hybrid' + ]; + rpc_name text; + commit_fn_def text; + required_indexes constant text[] := array[ + 'documents_title_trgm_idx', + 'document_chunks_content_trgm_idx', + 'document_labels_label_trgm_idx', + 'document_summaries_summary_trgm_idx', + 'document_chunks_embedding_hnsw_idx', + 'document_embedding_fields_embedding_hnsw_idx', + 'document_memory_cards_embedding_hnsw_idx', + 'documents_indexed_owner_title_idx', + 'document_table_facts_owner_document_page_idx', + 'document_embedding_fields_owner_chunk_idx', + 'document_index_units_owner_chunk_type_idx', + 'document_table_facts_source_image_idx', + 'document_pages_document_idx', + 'document_sections_document_idx', + 'document_chunks_document_idx', + 'document_memory_cards_document_idx', + 'document_embedding_fields_document_idx', + 'document_table_facts_document_idx', + 'document_index_units_document_idx', + 'rag_retrieval_logs_owner_created_idx', + 'rag_retrieval_logs_miss_idx', + 'rag_retrieval_logs_strategy_idx', + 'documents_search_idx', + 'documents_registry_projection_lookup_idx', + 'document_chunks_anchor_idx', + 'document_embedding_fields_search_tsv_chunk_gin_idx', + 'document_index_units_heading_path_idx', + 'document_index_units_search_idx', + 'document_index_units_terms_idx', + 'document_memory_cards_search_idx' + ]; + index_aliases constant jsonb := jsonb_build_object( + 'documents_title_trgm_idx', jsonb_build_array('documents_title_search_tsv_idx', 'documents_title_search_idx'), + 'document_chunks_content_trgm_idx', jsonb_build_array('document_chunks_search_tsv_idx', 'document_chunks_search_idx'), + 'document_table_facts_owner_document_page_idx', jsonb_build_array('document_table_facts_owner_idx'), + 'document_pages_document_idx', jsonb_build_array('document_pages_document_id_page_number_key'), + 'document_sections_document_idx', jsonb_build_array('document_sections_document_id_idx'), + 'rag_retrieval_logs_owner_created_idx', jsonb_build_array('rag_retrieval_logs_owner_id_idx') + ); +begin + select t.oid, n.nspname + into vector_type_oid, vector_schema + from pg_type t + join pg_namespace n on n.oid = t.typnamespace + where t.typname = 'vector' + and n.nspname = 'extensions' + limit 1; + + if vector_type_oid is null then + missing := array_append(missing, 'extensions.vector_type'); + end if; + + if to_regprocedure('public.match_document_chunks(extensions.vector, integer, double precision, uuid, uuid)') is null then + missing := array_append(missing, 'match_document_chunks.extensions_vector_signature'); + end if; + if to_regprocedure('public.match_document_chunks_hybrid(extensions.vector, text, integer, double precision, uuid[], uuid)') is null then + missing := array_append(missing, 'match_document_chunks_hybrid.extensions_vector_signature'); + end if; + if to_regprocedure('public.match_document_chunks_text(text, integer, uuid[], uuid)') is null then + missing := array_append(missing, 'match_document_chunks_text.signature'); + end if; + if to_regprocedure('public.match_document_lookup_chunks_text(text, uuid[], integer, uuid)') is null then + missing := array_append(missing, 'match_document_lookup_chunks_text.signature'); + end if; + if to_regprocedure('public.match_document_memory_cards_hybrid(extensions.vector, text, integer, double precision, uuid[], uuid)') is null then + missing := array_append(missing, 'match_document_memory_cards_hybrid.extensions_vector_signature'); + end if; + if to_regprocedure('public.match_document_memory_cards_hybrid_v2(extensions.vector, text, integer, double precision, uuid[], uuid)') is null then + missing := array_append(missing, 'match_document_memory_cards_hybrid_v2.extensions_vector_signature'); + end if; + if to_regprocedure('public.match_document_index_units_hybrid(extensions.vector, text, integer, double precision, uuid[], uuid)') is null then + missing := array_append(missing, 'match_document_index_units_hybrid.extensions_vector_signature'); + end if; + if to_regprocedure('public.match_document_embedding_fields_hybrid(extensions.vector, text, integer, double precision, uuid[], uuid)') is null then + missing := array_append(missing, 'match_document_embedding_fields_hybrid.extensions_vector_signature'); + end if; + if to_regprocedure('public.match_documents_for_query(text, integer, uuid)') is null then + missing := array_append(missing, 'match_documents_for_query.signature'); + end if; + if to_regprocedure('public.match_document_table_facts_text(text, integer, uuid[], uuid)') is null then + missing := array_append(missing, 'match_document_table_facts_text.signature'); + end if; + if to_regprocedure('public.explain_retrieval_rpc(text, text, integer, uuid, uuid[], boolean)') is null then + missing := array_append(missing, 'explain_retrieval_rpc.signature'); + end if; + if to_regclass('public.rag_retrieval_logs') is null then + missing := array_append(missing, 'rag_retrieval_logs.table'); + end if; + + foreach index_name in array required_indexes loop + if not exists ( + select 1 + from pg_class c + join pg_namespace ns on ns.oid = c.relnamespace + where ns.nspname = 'public' + and c.relname = index_name + and c.relkind = 'i' + ) + and not ( + index_aliases ? index_name + and exists ( + select 1 + from pg_class c + join pg_namespace ns on ns.oid = c.relnamespace + where ns.nspname = 'public' + and c.relkind = 'i' + and c.relname in ( + select jsonb_array_elements_text(index_aliases -> index_name) + ) + ) + ) then + missing := array_append(missing, index_name); + end if; + end loop; + + if vector_type_oid is not null then + zero_vec := (select ('[' || string_agg('0', ',') || ']') from generate_series(1, 1536))::extensions.vector(1536); + foreach rpc_name in array hybrid_rpcs loop + begin + execute format( + 'select 1 from public.%I($1, $2, 1, 0.1, null::uuid[], null::uuid) limit 1', + rpc_name + ) using zero_vec, probe_text; + exception + when undefined_function then + missing := array_append(missing, rpc_name || '.execution_signature'); + when others then + missing := array_append(missing, rpc_name || '.execution:' || SQLSTATE); + end; + end loop; + end if; + + commit_fn_def := pg_get_functiondef( + to_regprocedure( + 'public.commit_document_index_generation(uuid, uuid, text, integer, integer, integer, jsonb, jsonb, jsonb)' + ) + ); + if commit_fn_def is null then + missing := array_append(missing, 'commit_document_index_generation.signature'); + elsif position('from public.document_chunks replacement' in commit_fn_def) = 0 then + missing := array_append( + missing, + 'commit_document_index_generation.preserve_legacy_artifacts_migration' + ); + end if; + + select public.detect_legacy_ivfflat_indexes() into legacy_ivfflat_indexes; + + return jsonb_build_object( + 'ok', cardinality(missing) = 0, + 'missing', missing, + 'vector_extension_schema', vector_schema, + 'legacy_ivfflat_indexes', coalesce(legacy_ivfflat_indexes, array[]::text[]), + 'deferred_hnsw_indexes', array[]::text[], + 'checked_at', now() + ); +end; +$$; + +revoke execute on function public.search_schema_health() from public, anon, authenticated; +grant execute on function public.search_schema_health() to service_role; diff --git a/supabase/schema.sql b/supabase/schema.sql index 035fa14a89..78d6934f9a 100644 --- a/supabase/schema.sql +++ b/supabase/schema.sql @@ -3181,7 +3181,15 @@ declare 'document_index_units_document_idx', 'rag_retrieval_logs_owner_created_idx', 'rag_retrieval_logs_miss_idx', - 'rag_retrieval_logs_strategy_idx' + 'rag_retrieval_logs_strategy_idx', + 'documents_search_idx', + 'documents_registry_projection_lookup_idx', + 'document_chunks_anchor_idx', + 'document_embedding_fields_search_tsv_chunk_gin_idx', + 'document_index_units_heading_path_idx', + 'document_index_units_search_idx', + 'document_index_units_terms_idx', + 'document_memory_cards_search_idx' ]; -- Verified live equivalents: same table/column intent, different migration-era name. index_aliases constant jsonb := jsonb_build_object( diff --git a/supabase/search-health-unmonitored-indexes.json b/supabase/search-health-unmonitored-indexes.json index 41d8acb554..38b490e85e 100644 --- a/supabase/search-health-unmonitored-indexes.json +++ b/supabase/search-health-unmonitored-indexes.json @@ -1,18 +1,6 @@ { - "_comment": "Runtime index-monitoring ratchet (docs/database-drift-detection.md, 'Runtime index-monitoring ratchet'). Every repo-defined index on the retrieval-critical tables (documents, document_chunks, document_index_units, document_embedding_fields, document_memory_cards, rag_retrieval_logs) must be either monitored by search_schema_health() required_indexes/index_aliases or listed here with a reason and a disposition. `accepted-unmonitored`: absence would degrade an operational path but not clinical retrieval, and check:drift's full index inventory still reports it missing. `monitor-candidate`: retrieval-facing or currently absent on live (forensics section 1.3, 2026-08-14) — a Phase 4.4 migration extending required_indexes must decide it; it stays visibly flagged here until then. required_indexes changes travel by migration only, never by editing schema.sql. tests/search-health-index-coverage.test.ts enforces this file. Seeded 2026-08-18: 44 entries.", + "_comment": "Runtime index-monitoring ratchet (docs/database-drift-detection.md, 'Runtime index-monitoring ratchet'). Every repo-defined index on the retrieval-critical tables (documents, document_chunks, document_index_units, document_embedding_fields, document_memory_cards, rag_retrieval_logs) must be either monitored by search_schema_health() required_indexes/index_aliases or listed here with a reason and a disposition. `accepted-unmonitored`: absence would degrade an operational path but not clinical retrieval, and check:drift's full index inventory still reports it missing. `monitor-candidate`: retrieval-facing or currently absent on live (forensics section 1.3, 2026-08-14) — a Phase 4.4 migration extending required_indexes must decide it; it stays visibly flagged here until then. The disposition is currently unused: all eight 2026-08-18 candidates were decided by 20260819100300_monitor_restored_retrieval_indexes.sql and are now monitored, so they were removed from this file (the test rejects a listed index that is also monitored). It remains available for the next index this scope acquires. required_indexes changes travel by migration only, never by editing schema.sql. tests/search-health-index-coverage.test.ts enforces this file. Seeded 2026-08-18 with 44 entries; 2026-08-19 (Phase 4.4, ledger #316): the 8 monitor-candidates entered required_indexes, leaving 36 accepted-unmonitored.", "unmonitored": [ - { - "index": "documents_search_idx", - "table": "documents", - "disposition": "monitor-candidate", - "reason": "GIN over the documents search tsvector used by the title fast path; retrieval-facing but only the trgm title index is monitored today" - }, - { - "index": "documents_registry_projection_lookup_idx", - "table": "documents", - "disposition": "monitor-candidate", - "reason": "Absent on live per forensics section 1.3 (2026-08-14); registry-projection lookup path — Phase 4 restores it and Phase 4.4 decides required_indexes by migration" - }, { "index": "documents_owner_content_hash_unique_idx", "table": "documents", @@ -61,12 +49,6 @@ "disposition": "accepted-unmonitored", "reason": "Plain status index, codified into the chain by 20260818111000 (was schema.sql-only — drift backlog item 10, forensics 2.3 b); operational status scans" }, - { - "index": "document_chunks_anchor_idx", - "table": "document_chunks", - "disposition": "monitor-candidate", - "reason": "Absent on live per forensics section 1.3 (2026-08-14); anchor lookup for citation deep links — Phase 4 restores it and Phase 4.4 decides required_indexes by migration" - }, { "index": "document_chunks_content_hash_idx", "table": "document_chunks", @@ -133,12 +115,6 @@ "disposition": "accepted-unmonitored", "reason": "Rename target of the dropped document_embedding_fields_owner_idx, codified into the chain by 20260818111000 (was schema.sql-only — drift backlog item 10); owner FK support, superseded for retrieval by document_embedding_fields_owner_chunk_idx" }, - { - "index": "document_embedding_fields_search_tsv_chunk_gin_idx", - "table": "document_embedding_fields", - "disposition": "monitor-candidate", - "reason": "GIN over the embedding-field search tsvector, codified into the chain by 20260818111000 (was schema.sql-only — drift backlog item 10); lexical field retrieval path — decide required_indexes membership by migration in Phase 4.4 (present on production per the 2026-08-14 live-drift run)" - }, { "index": "document_embedding_fields_source_chunk_id_idx", "table": "document_embedding_fields", @@ -163,12 +139,6 @@ "disposition": "accepted-unmonitored", "reason": "Redundant with the monitored document_index_units_document_idx (same leading column); codified from live for FK support only" }, - { - "index": "document_index_units_heading_path_idx", - "table": "document_index_units", - "disposition": "monitor-candidate", - "reason": "Absent on live per forensics section 1.3 (2026-08-14); heading-path scoped lookups — Phase 4 restores it and Phase 4.4 decides required_indexes by migration" - }, { "index": "document_index_units_image_idx", "table": "document_index_units", @@ -199,12 +169,6 @@ "disposition": "accepted-unmonitored", "reason": "Producer-scoped generation bookkeeping for deep-memory commits; ingestion path" }, - { - "index": "document_index_units_search_idx", - "table": "document_index_units", - "disposition": "monitor-candidate", - "reason": "GIN over the index-unit search tsvector; lexical candidate gate for the index-unit hybrid RPC, retrieval-facing with no monitored equivalent (worst-covered table: 2 of 16 monitored)" - }, { "index": "document_index_units_source_chunk_id_idx", "table": "document_index_units", @@ -217,12 +181,6 @@ "disposition": "accepted-unmonitored", "reason": "source_image_id FK support codified from live; join/cleanup path" }, - { - "index": "document_index_units_terms_idx", - "table": "document_index_units", - "disposition": "monitor-candidate", - "reason": "GIN over normalized_terms used by index-unit lexical matching; retrieval-facing with no monitored equivalent" - }, { "index": "document_memory_cards_document_generation_idx", "table": "document_memory_cards", @@ -253,12 +211,6 @@ "disposition": "accepted-unmonitored", "reason": "Producer-scoped generation bookkeeping for deep-memory commits; ingestion path" }, - { - "index": "document_memory_cards_search_idx", - "table": "document_memory_cards", - "disposition": "monitor-candidate", - "reason": "GIN over the memory-card search tsvector; lexical half of the memory-card hybrid RPC, retrieval-facing with no monitored equivalent" - }, { "index": "document_memory_cards_section_idx", "table": "document_memory_cards",