diff --git a/AGENTS.md b/AGENTS.md index 8ae53dc041..721f3babc5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -415,6 +415,19 @@ Output-style plugins such as caveman mode may compress prose. They must never co - Bare-image storage scaffolding must discover its local schema owner at runtime and must never be reused as hosted migration SQL. - Run `npm run check:migration-role` after changing Supabase SQL, migration tooling, CI replay, or disaster-recovery instructions. - Run `npm run check:supabase-project` after changing Supabase env values. +- **Guard-migration contract.** Any mark-applied version, `supabase migration repair --status applied`, + hand-applied SQL later recorded as a migration, or other history repair MUST ship a fail-fast + validation guard migration in the same change, following `20260804110240_restore_rag_search_health_indexes.sql` + exactly (validates presence + `indisvalid`/`indisready` + normalized definition, never builds, + `set local` timeouts, one `raise exception`). `schema_drift_snapshot()` v2 (`20260818090000`) reports every + `supabase_migrations` version recorded without executed statements; `check:drift` fails on any such row + that lacks a reviewed `migration_history` entry in `supabase/drift-allowlist.json` pointing at its guard + (`guard.class` `validation` is mandatory for versions from 2026-08-18; `superseded`/`no_ddl` are for + pre-contract history only). Never allowlist a history row bare, and never widen an entry's class to make + it pass. Enforced offline by `tests/migration-history-guards.test.ts`; index-monitoring decisions on the + retrieval-critical tables are enforced by `tests/search-health-index-coverage.test.ts` + + `supabase/search-health-unmonitored-indexes.json` (`required_indexes` changes travel by migration only). + Full contract: `docs/database-drift-detection.md`. diff --git a/docs/audit/live-drift-forensics-2026-08.md b/docs/audit/live-drift-forensics-2026-08.md index 6005d58299..94f8b3e5e8 100644 --- a/docs/audit/live-drift-forensics-2026-08.md +++ b/docs/audit/live-drift-forensics-2026-08.md @@ -274,3 +274,32 @@ The A1/S1 packet must re-verify `generation_quality_gate:*` dominance on healthy choosing any code mitigation. Residual: hybrid fan-out still costs ~8.5 s worst-observed — owned by the remaining remediation phases, not a route-budget change (`#231`'s stop condition stands). `check:production-readiness` on the final state is **pending**. + +## Phase 6 — Future-proofing (repo-side; one migration authored, NOT deployed) + +_2026-08-18 (repo-only session; no hosted read or mutation)._ Built per plan §6.1–6.3, worker chat +without a ledger row (residual queued via `npm run issues:add`, not on `#316`): + +- **6.1 History-integrity probe.** `20260818090000_schema_drift_snapshot_history_probe.sql` redefines + `public.schema_drift_snapshot()` (v2) to also return `migration_history` — every + `supabase_migrations.schema_migrations` version with `statements IS NULL` or empty — plus + `migration_history_probe` (`ok` / `no_history_table` / `no_statements_column`). Mirrored into + `schema.sql`; `drift-manifest.json` regenerated (Docker replay executed the new body: probe + `no_history_table`, `snapshot_version` 2). `check:drift` reports each live row as + `! [migration_history] no_statements ` unless a validated allowlist entry covers it. + **Not deployed** — needs the owner-approved production migration window (approval map, Phase 6.1, + after Phase 4). Until then the live run shows the `schema_drift_snapshot()` function `def_hash` + mismatch (repo-ahead) and an info line naming the pending deploy. +- **6.2 Guard-migration contract.** Written into `docs/database-drift-detection.md` and `AGENTS.md` + ("Supabase project safety"). Allowlist `migration_history` entries carry `guard {class, migration, +objects}`; classes `validation` (mandatory from 2026-08-18), `superseded`, `no_ddl`. + `tests/migration-history-guards.test.ts` verifies each guard file really covers its objects. Seeded + five §1.1 versions with repo-provable `superseded` guards (`20260701010000`, `20260701020000`, + `20260701030000`, `20260701060000`, `20260702000000`); the remaining §1.1 rows and the 2026-07-12 + batch are deliberately **not** allowlisted and are the expected findings of the first post-deploy run. +- **6.3 Runtime coverage ratchet.** `tests/search-health-index-coverage.test.ts` + + `supabase/search-health-unmonitored-indexes.json`: every repo-defined index on the six + retrieval-critical tables is monitored or listed with reason + disposition. Failed with exactly 44 + names before the list existed; passes with 44 entries (8 `monitor-candidate`, including the three + §1.3-absent indexes on those tables: `document_chunks_anchor_idx`, + `document_index_units_heading_path_idx`, `documents_registry_projection_lookup_idx`). diff --git a/docs/branch-review-records/014fee0f5a9c0fc049bfa0047afa855ccd498160d291ebb5c3d2eb0b0749a7cf.record.md b/docs/branch-review-records/014fee0f5a9c0fc049bfa0047afa855ccd498160d291ebb5c3d2eb0b0749a7cf.record.md new file mode 100644 index 0000000000..a61cdd0316 --- /dev/null +++ b/docs/branch-review-records/014fee0f5a9c0fc049bfa0047afa855ccd498160d291ebb5c3d2eb0b0749a7cf.record.md @@ -0,0 +1 @@ +| 2026-08-17 | claude/database-drift-allowlist-48839e | ccbb4cd8a95fd074f70931f73e79b31b525ef579 | remediation Phase 6: schema_drift_snapshot v2 history probe migration (not deployed), check-drift migration_history allowlist + guard contract, search-health index-monitoring ratchet, PR #2058 | self-review: migration authored + mirrored + manifest regenerated, not deployed; five superseded guards seeded, rest left as expected findings; 44-entry ratchet list red-then-green | check:migration-role pass; drift:manifest replay 46s; focused schema/drift suite 8 files 127 passed; verify:pr-local all gates green except tests/session-start-hook.test.ts (env: bash head 127 / temp EPERM, unrelated); check:rag:fixtures 36 golden cases pass | diff --git a/docs/database-drift-detection.md b/docs/database-drift-detection.md index b957463a8f..50e9710a50 100644 --- a/docs/database-drift-detection.md +++ b/docs/database-drift-detection.md @@ -1,6 +1,6 @@ # Database drift detection (`npm run check:drift`) -Last updated: 2026-07-10 +Last updated: 2026-08-18 (migration-history probe, guard-migration contract, index-monitoring ratchet — remediation plan Phase 6) This repo's worst operational incidents were live-vs-repo schema drift: hybrid retrieval RPCs silently broken on live for an unknown period, and migrations @@ -13,11 +13,12 @@ application-owned object against `supabase/schema.sql`. Three committed artifacts: -| Artifact | Role | -| -------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `supabase/migrations/20260706200000_schema_drift_snapshot.sql` | `public.schema_drift_snapshot()` — service-role-only RPC returning the normalized live inventory (also declared in `supabase/schema.sql`; a test enforces byte parity). | -| `supabase/drift-manifest.json` | The expected state: the same snapshot captured from a **from-scratch replay of `supabase/schema.sql`** into a disposable `supabase/postgres` Docker container (`npm run drift:manifest`). Embeds the sha256 of the schema.sql it came from. | -| `supabase/drift-allowlist.json` | Known, documented divergence (each entry has a `reason`). Reported as warnings; anything not listed fails the check. | +| Artifact | Role | +| ---------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `supabase/migrations/20260818090000_schema_drift_snapshot_history_probe.sql` | `public.schema_drift_snapshot()` v2 — service-role-only RPC returning the normalized live inventory plus the migration-history probe (supersedes `20260706200000`; also declared in `supabase/schema.sql`; a test enforces byte parity with the latest definer). | +| `supabase/drift-manifest.json` | The expected state: the same snapshot captured from a **from-scratch replay of `supabase/schema.sql`** into a disposable `supabase/postgres` Docker container (`npm run drift:manifest`). Embeds the sha256 of the schema.sql it came from. | +| `supabase/drift-allowlist.json` | Known, documented divergence (each entry has a `reason`; `migration_history` entries also need a `guard`). Reported as warnings; anything not listed fails the check. | +| `supabase/search-health-unmonitored-indexes.json` | The runtime index-monitoring ratchet: every repo-defined index on a retrieval-critical table that `search_schema_health()` does not monitor, with a reason and disposition (see below). | `npm run check:drift` (needs live service-role env) verifies the project ref, fails fast if the manifest is stale, calls the RPC, diffs, applies the @@ -78,6 +79,143 @@ Both are decisions rather than defects: adding validity to the snapshot RPC is a migration, and raising the cadence spends provider budget. Recorded so the gap is chosen, not assumed away. +A third limit was closed on the repo side by remediation-plan Phase 6 (2026-08-18) +and is live once migration `20260818090000` is deployed: **history repairs were +invisible.** The check compared object state only, so a `supabase_migrations` +version recorded without executed DDL stayed silent until its objects went +missing. The probe below turns that into a finding. + +## Migration-history probe + +`schema_drift_snapshot()` v2 (migration +`20260818090000_schema_drift_snapshot_history_probe.sql`; plan §6.1; +evidence `docs/audit/live-drift-forensics-2026-08.md` §1.1/§1.3) adds two keys: + +| Key | Value | +| ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `migration_history` | `[{version, name, signal}]` — every `supabase_migrations.schema_migrations` row where `statements IS NULL` (`signal: "null"`) or `cardinality(statements) = 0` (`"empty"`), ordered by version. | +| `migration_history_probe` | `"ok"`, `"no_history_table"` (the schema does not exist — true of every `drift:manifest` replay container), or `"no_statements_column"` (very old CLI history table). Never a silent `[]`. | + +Why the fingerprint matters: the CLI records the executed statements on every +`db push`; a row with none is a mark-applied / `migration repair --status +applied` / hand-applied version whose DDL the CLI never ran. §1.1 found the +2026-07-01…02 cluster and the 2026-07-12 batch in that state. §1.3 also found +migrations that **did** record executed DDL yet whose indexes are absent, so +the probe is a second signal beside the object inventory, never a replacement. + +How `check:drift` treats it (`scripts/check-drift.ts`): + +- The category is **never compared manifest-vs-live** — the manifest holds + `migration_history_probe: "no_history_table"` and `[]`, and the generic + category loop deliberately excludes it. Each live row is a finding of kind + `no_statements` unless a validated `migration_history` allowlist entry covers + that exact version. Output line: ` ! [migration_history] no_statements + :: (statements null|empty) — …`, captured by + `live-drift.yml`'s findings grep and routed into the pinned issue. +- A malformed entry (missing/unknown class, guard file absent, wrong ordering, + pre-contract class on a post-contract version) never matches: the row stays + a finding and the entry is printed under stale entries with the reasons. +- If the live payload lacks the key (probe not deployed), the run prints an + `info:` line and the function `def_hash` mismatch on `schema_drift_snapshot` + itself is the visible "deploy pending" signal — that is the ordinary + repo-ahead mechanism, not a special case. Deployment is a separately approved + production migration window (plan approval map, Phase 6.1, after Phase 4). + +**Expected first live run after deployment.** Only five versions are seeded in +the allowlist (below); the remaining §1.1 rows — `20260701040000 +drop_dead_drifted_hybrid_variants`, `20260702100000 +add_claim_ingestion_jobs_comment`, `20260702110000 drop_redundant_indexes`, +`20260702120000 rag_retrieval_logs_retention`, `20260702130000 +storage_cleanup_jobs_document_fk`, `20260702140000 +fix_reset_document_index_duplicate`, `20260702150000 +documents_owner_covering_index`, `20260702160000 fix_invoke_agent_url_to_guc`, +`20260702180000 promote_index_generation_id_columns`, and the 2026-07-12 batch +`20260712165915`…`20260712173000` — have **no repo-provable guard** and will be +reported as findings. That is the intended behaviour ("a history-repair row +without a validating guard migration becomes permanent, visible drift"); the +follow-up is to author fail-fast guard migrations for them (Phase 4.4 batches +cover the index ones) and allowlist each with a `validation` guard, not to +allowlist them bare. Versions the live probe does not report surface as stale +entries, which is how a wrong seed is caught. + +## Guard-migration contract + +**Rule (also in `AGENTS.md`, "Supabase project safety"): any mark-applied +version, `supabase migration repair --status applied`, hand-applied SQL that +is later recorded as a migration, or other history repair MUST ship a +fail-fast validation migration in the same change, following +`supabase/migrations/20260804110240_restore_rag_search_health_indexes.sql` +exactly.** Such a guard: + +- **validates, never builds** — no `create index`, no `create or replace` of + the objects it guards; it checks presence (`to_regclass` / + `to_regprocedure`), `pg_index.indisvalid AND indisready` for indexes, and a + normalized `pg_get_indexdef` / `pg_get_functiondef` match against the pinned + canonical definition; +- uses `set local lock_timeout` / `set local statement_timeout` (never bare + `set`), and raises one `raise exception … Missing: %; Invalid: %; +Mismatched: %` naming every failure; +- is marked applied only after the live validation passes, and its file name + is what the allowlist entry points at. + +Allowlist entry shape (`supabase/drift-allowlist.json`): + +```json +{ + "category": "migration_history", + "kind": "no_statements", + "key": "<14-digit version>", + "reason": "why the row has no executed DDL (mark-applied after prebuild, repair, rename …) — > 20 chars", + "guard": { "class": "validation", "migration": "_.sql", "objects": ["", "…"] } +} +``` + +| `guard.class` | Meaning | Machine check (`check:drift` structural + `tests/migration-history-guards.test.ts` object-level) | +| ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | +| `validation` | A later fail-fast guard migration proves the listed objects. **Required for every version ≥ `20260818000000`.** | file exists, version > key, contains `raise exception`, creates no index, mentions every `objects` name; `objects` non-empty | +| `superseded` | A later migration re-created every listed object with recorded statements (squashed baseline, renumber, hotfix later codified). Pre-contract history only. | file exists, version > key, `create … ` for every listed object in the guard file **and** in the version's own file; key < contract date | +| `no_ddl` | The version's own file has no effect (comments only / `select 1;` placeholder). | `guard.migration` is the version's own file; stripped body is empty or `select 1;` | + +Retiring an entry: when a version is genuinely re-recorded with statements (or +history is squashed and the row disappears) the entry shows as stale on the +next run — delete it. Never widen a class or drop `objects` to make an entry +pass; the finding is the point. + +## Runtime index-monitoring ratchet + +`search_schema_health()` monitors a curated `required_indexes` list (22 names +in the latest definer, `20260706010000_search_schema_health_m13_guard.sql`) +plus `index_aliases`; the 20 indexes absent on live in 2026-08 were invisible +to it. `tests/search-health-index-coverage.test.ts` now requires that **every +repo-defined index on the retrieval-critical tables** — `documents`, +`document_chunks`, `document_index_units`, `document_embedding_fields`, +`document_memory_cards`, `rag_retrieval_logs` — is either monitored (in +`required_indexes` or an alias value) or listed in +`supabase/search-health-unmonitored-indexes.json` with a `reason` (> 20 chars) +and a `disposition`: + +- `accepted-unmonitored` — absence would degrade an operational path + (ingestion bookkeeping, FK support, listings) but not clinical retrieval; + `check:drift`'s full index inventory still reports it missing. +- `monitor-candidate` — retrieval-facing (`*_search_idx` / `*_terms_idx` GINs) + or currently absent on live per forensics §1.3 (`document_chunks_anchor_idx`, + `document_index_units_heading_path_idx`, + `documents_registry_projection_lookup_idx` — the three of the 20 that sit on + these tables; the other 17 are on tables outside this scope). A Phase 4.4 + migration extending `required_indexes` must decide each; they stay flagged + until then. + +"Repo-defined" is computed two ways and unioned — an order-aware replay of +every `create/drop index` in `supabase/migrations/`, and the manifest's +`snapshot.indexes` — so a schema.sql-only index (drift backlog item 10) or a +migration-only one cannot hide. Constraint-backed `*_pkey` indexes are exempt +(the constraint inventory guards them). The list is also checked for stale +entries (index no longer defined, or now monitored) and duplicates. +`required_indexes` changes travel by migration only, never by editing +`schema.sql`; the test cross-checks the two copies. Seeded 2026-08-18 with 44 +entries (8 monitor candidates); the test failed with exactly those 44 names +before the list existed. + ### Workflow - Change `supabase/schema.sql` → run `npm run drift:manifest` (Docker) in the diff --git a/docs/outstanding-issues-inbox/d6ce8a1d-518d-48fc-8a9a-7796f060a46e.json b/docs/outstanding-issues-inbox/d6ce8a1d-518d-48fc-8a9a-7796f060a46e.json new file mode 100644 index 0000000000..de855efc7d --- /dev/null +++ b/docs/outstanding-issues-inbox/d6ce8a1d-518d-48fc-8a9a-7796f060a46e.json @@ -0,0 +1,14 @@ +{ + "version": 2, + "id": "d6ce8a1d-518d-48fc-8a9a-7796f060a46e", + "createdOn": "2026-08-17", + "action": "add", + "payload": { + "pri": "P2", + "type": "task", + "summary": "Deploy the 20260818090000 schema_drift_snapshot v2 history probe (Phase 6.1) in an approved production window after Phase 4, triage the first unguarded no-statements report, and author fail-fast guard migrations for the pre-contract 2026-07-01..02 and 2026-07-12 rows", + "detail": "Repo side of remediation plan Phase 6 landed (probe migration, guard-migration contract in docs/database-drift-detection.md + AGENTS.md, tests/migration-history-guards.test.ts, tests/search-health-index-coverage.test.ts + supabase/search-health-unmonitored-indexes.json). The migration is NOT deployed: it needs the owner-approved production migration deploy window (plan approval map, Phase 6.1, after Phase 4). Expected first live run: the ~9 remaining section 1.1 versions plus the 2026-07-12 batch (20260712165915..20260712173000) are reported as unguarded no_statements findings because no repo-provable guard exists; each needs a validation guard migration (20260804110240 pattern) + a migration_history allowlist entry, not a bare allowlist. Also decide the 8 monitor-candidate indexes in the ratchet file by a required_indexes migration (Phase 4.4). Do not touch #316/#056 for this; those rows are owned by the Phase 1.2 / Phase 2 sessions.", + "source": "Phase 6 worker chat 2026-08-18; docs/database-remediation-plan.md section 6; docs/audit/live-drift-forensics-2026-08.md Phase 6", + "issueUlid": "01M089RSB6Q5JHBJX1MSZ0DR85" + } +} diff --git a/scripts/check-drift.ts b/scripts/check-drift.ts index a6b7a3004b..8240dda129 100644 --- a/scripts/check-drift.ts +++ b/scripts/check-drift.ts @@ -1,5 +1,5 @@ import { createHash } from "node:crypto"; -import { readFileSync, writeFileSync } from "node:fs"; +import { existsSync, readFileSync, writeFileSync } from "node:fs"; import { loadEnvConfig } from "@next/env"; loadEnvConfig(process.cwd()); @@ -21,27 +21,106 @@ loadEnvConfig(process.cwd()); * Known, documented divergence is carried in supabase/drift-allowlist.json — * every entry needs a reason and is reported as a warning, never silently * dropped. Anything not allowlisted exits 1. See docs/database-drift-detection.md. + * + * Migration-history probe (snapshot v2, migration 20260818090000): the live + * snapshot also carries `migration_history` — every + * supabase_migrations.schema_migrations version whose `statements` is NULL or + * empty, i.e. a history repair / mark-applied row whose DDL the CLI never + * executed. That category is never compared manifest-vs-live (the replay + * container has no history table); each live row is a finding unless a + * reviewed `migration_history` allowlist entry points at a real guard + * migration (see docs/database-drift-detection.md "Guard-migration contract"). */ type SnapshotObject = Record; type Snapshot = Record; -type AllowlistEntry = { +export const HISTORY_PROBE_MIGRATION = "20260818090000_schema_drift_snapshot_history_probe.sql"; +/** Versions at or after this date must use the `validation` guard class. */ +export const GUARD_CONTRACT_VERSION = "20260818000000"; +export const HISTORY_GUARD_CLASSES = ["validation", "superseded", "no_ddl"] as const; +export type HistoryGuardClass = (typeof HISTORY_GUARD_CLASSES)[number]; + +export type HistoryGuard = { + class: HistoryGuardClass; + migration: string; + objects?: string[]; +}; + +export type AllowlistEntry = { category: string; - kind: "missing_live" | "unexpected_live" | "mismatch" | "alias"; + kind: "missing_live" | "unexpected_live" | "mismatch" | "alias" | "no_statements"; key: string; live_key?: string; reason: string; + guard?: HistoryGuard; }; -type Finding = { +export type Finding = { category: string; - kind: "missing_live" | "unexpected_live" | "mismatch"; + kind: "missing_live" | "unexpected_live" | "mismatch" | "no_statements"; key: string; detail?: string; }; +type MigrationHistoryRow = { version: string; name?: string; signal?: string }; + const read = (relative: string) => readFileSync(new URL(`../${relative}`, import.meta.url), "utf8"); +const migrationExists = (fileName: string) => + existsSync(new URL(`../supabase/migrations/${fileName}`, import.meta.url)); + +const MIGRATION_FILE = /^(\d{14})_.+\.sql$/; + +/** + * Structural validation of a `migration_history` allowlist entry. A malformed + * entry never matches a finding, so it can never silence a history row: the + * row stays an unexpected finding and the entry is reported as stale. + * Object-level checks (does the guard file really validate the named objects) + * live in tests/migration-history-guards.test.ts. + */ +export function historyEntryProblems( + entry: AllowlistEntry, + options: { migrationExists?: (fileName: string) => boolean } = {}, +): string[] { + const exists = options.migrationExists ?? migrationExists; + const problems: string[] = []; + if (entry.category !== "migration_history") problems.push("category must be migration_history"); + if (entry.kind !== "no_statements") problems.push("kind must be no_statements"); + if (!/^\d{14}$/.test(entry.key)) problems.push("key must be a 14-digit migration version"); + if (typeof entry.reason !== "string" || entry.reason.trim().length <= 20) { + problems.push("reason must be a real explanation (> 20 chars)"); + } + const guard = entry.guard; + if (!guard || typeof guard !== "object") { + problems.push("guard is required"); + return problems; + } + if (!HISTORY_GUARD_CLASSES.includes(guard.class)) { + problems.push(`guard.class must be one of ${HISTORY_GUARD_CLASSES.join("|")}`); + } + const fileMatch = typeof guard.migration === "string" ? guard.migration.match(MIGRATION_FILE) : null; + if (!fileMatch) { + problems.push("guard.migration must be a <14-digit version>_.sql file name"); + return problems; + } + if (!exists(guard.migration)) + problems.push(`guard.migration ${guard.migration} does not exist under supabase/migrations/`); + const guardVersion = fileMatch[1]; + if (guard.class === "no_ddl") { + if (guardVersion !== entry.key) problems.push("no_ddl guard must be the version's own migration file"); + } else if (!(guardVersion > entry.key)) { + problems.push(`${guard.class} guard must be a later migration than ${entry.key}`); + } + if (guard.class === "validation" && (!Array.isArray(guard.objects) || guard.objects.length === 0)) { + problems.push("validation guard must list the objects it proves (guard.objects)"); + } + if (guard.class !== "validation" && entry.key >= GUARD_CONTRACT_VERSION) { + problems.push( + `versions from ${GUARD_CONTRACT_VERSION} must use a validation guard (guard-migration contract); ${guard.class} is for pre-contract history only`, + ); + } + return problems; +} export function normalizedSchemaSha256(schemaSqlText: string) { return createHash("sha256").update(schemaSqlText.replace(/\r\n/g, "\n")).digest("hex"); @@ -127,6 +206,7 @@ export function compareDriftSnapshots( expected: Snapshot, live: Snapshot, allowlist: AllowlistEntry[], + options: { historyEntryProblems?: (entry: AllowlistEntry) => string[] } = {}, ): DriftComparison { const findings: Finding[] = []; const infos: string[] = []; @@ -158,6 +238,35 @@ export function compareDriftSnapshots( } } + // Migration-history probe (snapshot v2). Never compared manifest-vs-live — + // the manifest replay has no supabase_migrations schema — so every live row + // recorded without executed statements is a finding unless a validated + // `migration_history` allowlist entry covers it. + const historyRows = live.migration_history; + if (historyRows === undefined) { + infos.push( + `migration-history probe not present in the live snapshot — migration ${HISTORY_PROBE_MIGRATION} is not deployed; ` + + `the schema_drift_snapshot() function mismatch is that pending deploy, not a body regression`, + ); + } else if (live.migration_history_probe !== "ok") { + infos.push( + `migration-history probe reported '${String(live.migration_history_probe)}' — history rows were not inspected`, + ); + } else if (Array.isArray(historyRows)) { + for (const raw of historyRows) { + if (!raw || typeof raw !== "object") continue; + const row = raw as MigrationHistoryRow; + findings.push({ + category: "migration_history", + kind: "no_statements", + key: String(row.version), + detail: + `${row.name ?? "(unnamed)"} (statements ${row.signal ?? "null"}) — history row recorded without executed DDL; ` + + `needs a fail-fast guard migration + reviewed allowlist entry`, + }); + } + } + // Apply the allowlist. Alias entries assert the live database carries the // same index under a legacy name; they consume both the missing_live finding // for the manifest name and the unexpected_live finding for the legacy name. @@ -184,11 +293,24 @@ export function compareDriftSnapshots( return finding.kind === "unexpected_live" && finding.key === entry.live_key; }; + // A migration_history entry only counts when it is structurally valid and + // points at a real guard migration; a malformed entry can never silence a + // history row (the finding stays and the entry is reported as stale). + const historyProblems = options.historyEntryProblems ?? historyEntryProblems; + const matchesHistory = (entry: AllowlistEntry, finding: Finding): boolean => + finding.category === "migration_history" && + finding.kind === "no_statements" && + entry.category === "migration_history" && + entry.kind === "no_statements" && + entry.key === finding.key && + historyProblems(entry).length === 0; + for (const finding of findings) { - const entry = allowlist.find( - (candidate) => - (candidate.kind === "alias" && matchesAlias(candidate, finding)) || - (candidate.kind === finding.kind && candidate.category === finding.category && candidate.key === finding.key), + const entry = allowlist.find((candidate) => + finding.category === "migration_history" + ? matchesHistory(candidate, finding) + : (candidate.kind === "alias" && matchesAlias(candidate, finding)) || + (candidate.kind === finding.kind && candidate.category === finding.category && candidate.key === finding.key), ); if (entry) { usedEntries.add(entry); @@ -293,7 +415,10 @@ async function main() { if (staleEntries.length > 0) { console.log(`\nStale allowlist entries (${staleEntries.length}) — no longer matching, remove them:`); for (const entry of staleEntries) { - console.log(` ? [${entry.category}] ${entry.kind} ${entry.key}`); + const problems = entry.category === "migration_history" ? historyEntryProblems(entry) : []; + console.log( + ` ? [${entry.category}] ${entry.kind} ${entry.key}${problems.length ? ` :: invalid entry — ${problems.join("; ")}` : ""}`, + ); } } if (remaining.length > 0) { diff --git a/supabase/drift-allowlist.json b/supabase/drift-allowlist.json index baa76fbdc6..c453ac2fb5 100644 --- a/supabase/drift-allowlist.json +++ b/supabase/drift-allowlist.json @@ -1,4 +1,65 @@ { - "_comment": "Known live-vs-schema.sql divergence. Empty after the forward-only 2026-07-13 reconciliation. Every future entry must carry a specific reason; check:drift fails on anything not listed here.", - "entries": [] + "_comment": "Known live-vs-schema.sql divergence. Object categories (indexes, functions, ...) have been empty since the forward-only 2026-07-13 reconciliation. Every entry must carry a specific reason; check:drift fails on anything not listed here.", + "_migration_history_contract": "Entries with category `migration_history` / kind `no_statements` cover supabase_migrations.schema_migrations versions recorded WITHOUT executed statements (history repair / mark-applied rows) that public.schema_drift_snapshot() v2 (migration 20260818090000) reports. Each needs a `reason` and a `guard` = {class, migration, objects}. class `validation`: a later fail-fast validation migration (20260804110240 pattern) proves the objects; required for every version from 20260818000000. class `superseded`: a later migration re-created every listed object with recorded statements (pre-contract history only). class `no_ddl`: the version's own file has no effect (comments / `select 1;`). check:drift ignores a malformed entry (the row stays a finding) and tests/migration-history-guards.test.ts verifies the guard file really covers the listed objects. Seeded 2026-08-18 from docs/audit/live-drift-forensics-2026-08.md section 1.1 (versions resolved by name from the repo, not by a live read); an entry whose version is not reported by the live probe surfaces as a stale entry. See docs/database-drift-detection.md.", + "entries": [ + { + "category": "migration_history", + "kind": "no_statements", + "key": "20260701010000", + "reason": "2026-07-01 hybrid-retrieval hotfix applied to live by raw SQL and mark-applied (forensics section 1.1); match_document_chunks_hybrid has since been redefined by later migrations with recorded statements and its body/ACL sit in the drift inventory", + "guard": { + "class": "superseded", + "migration": "20260714110000_promote_documents_index_generation_id.sql", + "objects": ["match_document_chunks_hybrid"] + } + }, + { + "category": "migration_history", + "kind": "no_statements", + "key": "20260701020000", + "reason": "2026-07-01 hybrid-retrieval hotfix applied to live by raw SQL and mark-applied (forensics section 1.1); all three RPCs were re-codified verbatim from live by 20260701140631 and sit in the drift inventory", + "guard": { + "class": "superseded", + "migration": "20260701140631_codify_live_retrieval_rpcs.sql", + "objects": [ + "match_document_index_units_hybrid", + "match_document_embedding_fields_hybrid", + "match_document_memory_cards_hybrid_v2" + ] + } + }, + { + "category": "migration_history", + "kind": "no_statements", + "key": "20260701030000", + "reason": "2026-07-01 search_schema_health execution-smoke revision mark-applied after hand-apply (forensics section 1.1); search_schema_health() was redefined by the M13 guard migration and its def_hash sits in the drift inventory", + "guard": { + "class": "superseded", + "migration": "20260706010000_search_schema_health_m13_guard.sql", + "objects": ["search_schema_health"] + } + }, + { + "category": "migration_history", + "kind": "no_statements", + "key": "20260701060000", + "reason": "2026-07-01 clinical query-term corrector mark-applied after hand-apply (forensics section 1.1); correct_clinical_query_terms was redefined by the public-title corrector migration and sits in the drift inventory", + "guard": { + "class": "superseded", + "migration": "20260717171000_public_title_corrector.sql", + "objects": ["correct_clinical_query_terms"] + } + }, + { + "category": "migration_history", + "kind": "no_statements", + "key": "20260702000000", + "reason": "2026-07-02 commit_document_index_generation legacy-artifact fix mark-applied after hand-apply (forensics section 1.1); the function was redefined by the generation-commit fence migration and sits in the drift inventory", + "guard": { + "class": "superseded", + "migration": "20260713062125_fence_index_generation_commit.sql", + "objects": ["commit_document_index_generation"] + } + } + ] } diff --git a/supabase/drift-manifest.json b/supabase/drift-manifest.json index a27ecf2477..20cb6ea453 100644 --- a/supabase/drift-manifest.json +++ b/supabase/drift-manifest.json @@ -1,9 +1,9 @@ { - "generated_at": "2026-08-16T14:37:41.042Z", + "generated_at": "2026-08-17T16:38:39.818Z", "generator": "scripts/generate-drift-manifest.ts", "postgres_image": "supabase/postgres:17.6.1.127@sha256:be60aee15997daca475b710b734bc6bfe52cd544dcd7e9fd2ff58210b6747d83", - "schema_sha256": "365e3368a47b3ab725807d22bf9915e733fc60fd583abe9ebed92f4d2b9776d7", - "replay_seconds": 12, + "schema_sha256": "a6fb923400f8566966bc3387f7e58c0b322aed4ae8ff7daa20f9eb5b53567b42", + "replay_seconds": 46, "snapshot": { "views": [ { @@ -7320,7 +7320,7 @@ "postgres=X/postgres", "service_role=X/postgres" ], - "def_hash": "903e3420eb672a0fb4f5e608c0b4726e", + "def_hash": "462be6fc6532cc3bda833484ebd72384", "signature": "public.schema_drift_snapshot()" }, { @@ -8288,6 +8288,8 @@ ] } ], - "snapshot_version": 1 + "snapshot_version": 2, + "migration_history": [], + "migration_history_probe": "no_history_table" } } diff --git a/supabase/migrations/20260818090000_schema_drift_snapshot_history_probe.sql b/supabase/migrations/20260818090000_schema_drift_snapshot_history_probe.sql new file mode 100644 index 0000000000..d0e0095d45 --- /dev/null +++ b/supabase/migrations/20260818090000_schema_drift_snapshot_history_probe.sql @@ -0,0 +1,200 @@ +-- schema_drift_snapshot() v2: add the migration-history integrity probe. +-- +-- Supersedes 20260706200000_schema_drift_snapshot.sql (same inventory, two new +-- keys). Plan of record: docs/database-remediation-plan.md phase 6.1; evidence +-- docs/audit/live-drift-forensics-2026-08.md sections 1.1 and 1.3. +-- +-- WHY: supabase_migrations.schema_migrations rows whose `statements` column is +-- NULL or empty are the fingerprint of a history repair / mark-applied version — +-- a version recorded as applied whose DDL was never executed by the CLI (the +-- 2026-07-01..02 cluster and the 2026-07-12 batch in section 1.1). Until now the +-- drift check deliberately ignored history and compared object state only, so a +-- history repair without a validating guard migration was invisible until its +-- objects went missing. This probe returns those versions so `npm run check:drift` +-- can report every one that is not covered by a reviewed, guard-backed allowlist +-- entry (supabase/drift-allowlist.json, category `migration_history`). Section +-- 1.3 also shows migrations that DID record executed DDL yet whose indexes are +-- absent, so this probe is a second signal beside the object inventory, never a +-- replacement for it. +-- +-- What changed vs v1: +-- * `snapshot_version` 1 -> 2. +-- * `migration_history`: jsonb array of {version, name, signal} for every row of +-- supabase_migrations.schema_migrations where statements IS NULL ('null') or +-- cardinality(statements) = 0 ('empty'), ordered by version. +-- * `migration_history_probe`: 'ok' | 'no_history_table' | 'no_statements_column'. +-- The drift-manifest replay container (`npm run drift:manifest`) has no +-- supabase_migrations schema, so the manifest records 'no_history_table' and an +-- empty array; check:drift never compares this category manifest-vs-live — it +-- compares live-vs-allowlist. Very old CLI history tables lack `statements` +-- (reported honestly as 'no_statements_column' rather than as "clean"). +-- * Read via dynamic SQL because search_path is pinned to '' and the schema may +-- not exist on replay (same pattern as the storage.buckets block). +-- +-- Read-only, security definer, service_role execute only, exactly like v1. +-- Deployment of this migration to the live project is a separately approved +-- window; until it is applied, check:drift reports the function `def_hash` +-- mismatch for schema_drift_snapshot itself and prints an info line that the +-- probe is not yet available on live. + +create or replace function public.schema_drift_snapshot() +returns jsonb +language plpgsql +stable +security definer +set search_path to '' +as $$ +declare + snapshot jsonb; + buckets jsonb := '[]'::jsonb; + history jsonb := '[]'::jsonb; + history_probe text := 'no_history_table'; +begin + select jsonb_build_object( + 'snapshot_version', 2, + 'captured_at', now(), + 'extensions', coalesce(( + select jsonb_agg(jsonb_build_object('name', e.extname, 'schema', n.nspname) order by e.extname) + from pg_extension e + join pg_namespace n on n.oid = e.extnamespace + where e.extname <> 'plpgsql' + ), '[]'::jsonb), + 'tables', coalesce(( + select jsonb_agg(jsonb_build_object( + 'name', c.relname, + 'rls_enabled', c.relrowsecurity, + 'rls_forced', c.relforcerowsecurity, + 'reloptions', (select array_agg(o.opt order by o.opt) from unnest(c.reloptions) o(opt)), + 'acl', (select array_agg(a.item::text order by a.item::text) from unnest(coalesce(c.relacl, acldefault('r', c.relowner))) a(item)), + 'columns', ( + select jsonb_agg(jsonb_build_object( + 'name', att.attname, + 'type', format_type(att.atttypid, att.atttypmod), + 'not_null', att.attnotnull, + 'identity', att.attidentity, + 'generated', att.attgenerated, + 'default', pg_get_expr(ad.adbin, ad.adrelid) + ) order by att.attname) + from pg_attribute att + left join pg_attrdef ad on ad.adrelid = att.attrelid and ad.adnum = att.attnum + where att.attrelid = c.oid and att.attnum > 0 and not att.attisdropped + ) + ) order by c.relname) + from pg_class c + where c.relnamespace = 'public'::regnamespace and c.relkind = 'r' + ), '[]'::jsonb), + 'views', coalesce(( + select jsonb_agg(jsonb_build_object( + 'name', c.relname, + 'def_hash', md5(regexp_replace(pg_get_viewdef(c.oid), '\s+', '', 'g')) + ) order by c.relname) + from pg_class c + where c.relnamespace = 'public'::regnamespace and c.relkind in ('v', 'm') + ), '[]'::jsonb), + 'functions', coalesce(( + select jsonb_agg(jsonb_build_object( + 'signature', p.oid::regprocedure::text, + 'def_hash', md5(regexp_replace(regexp_replace(regexp_replace(pg_get_functiondef(p.oid), '/\*.*?\*/', '', 'gs'), '--[^\n]*', '', 'g'), '\s+', '', 'g')), + 'acl', (select array_agg(a.item::text order by a.item::text) from unnest(coalesce(p.proacl, acldefault('f', p.proowner))) a(item)) + ) order by p.oid::regprocedure::text) + from pg_proc p + where p.pronamespace = 'public'::regnamespace + and p.prokind = 'f' + and not exists ( + select 1 from pg_depend dep + where dep.classid = 'pg_proc'::regclass and dep.objid = p.oid and dep.deptype = 'e' + ) + ), '[]'::jsonb), + 'indexes', coalesce(( + select jsonb_agg(jsonb_build_object( + 'name', ci.relname, + 'table', ct.relname, + 'def', pg_get_indexdef(ci.oid), + 'def_hash', md5(regexp_replace(pg_get_indexdef(ci.oid), '\s+', '', 'g')) + ) order by ci.relname) + from pg_index i + join pg_class ci on ci.oid = i.indexrelid + join pg_class ct on ct.oid = i.indrelid + where ct.relnamespace = 'public'::regnamespace + and ci.relnamespace = 'public'::regnamespace + ), '[]'::jsonb), + 'policies', coalesce(( + select jsonb_agg(jsonb_build_object( + 'schema', pol.schemaname, + 'table', pol.tablename, + 'name', pol.policyname, + 'permissive', pol.permissive, + 'roles', (select array_agg(r.role::text order by r.role::text) from unnest(pol.roles) r(role)), + 'cmd', pol.cmd, + 'qual', pol.qual, + 'with_check', pol.with_check + ) order by pol.schemaname, pol.tablename, pol.policyname) + from pg_policies pol + where pol.schemaname in ('public', 'storage') + ), '[]'::jsonb), + 'constraints', coalesce(( + select jsonb_agg(jsonb_build_object( + 'table', ct.relname, + 'name', con.conname, + 'def', pg_get_constraintdef(con.oid) + ) order by ct.relname, con.conname) + from pg_constraint con + join pg_class ct on ct.oid = con.conrelid + where con.connamespace = 'public'::regnamespace and ct.relkind = 'r' + ), '[]'::jsonb), + 'triggers', coalesce(( + select jsonb_agg(jsonb_build_object( + 'table', ct.relname, + 'name', t.tgname, + 'def', pg_get_triggerdef(t.oid) + ) order by ct.relname, t.tgname) + from pg_trigger t + join pg_class ct on ct.oid = t.tgrelid + where ct.relnamespace = 'public'::regnamespace and not t.tgisinternal + ), '[]'::jsonb) + ) into snapshot; + + if to_regclass('storage.buckets') is not null then + execute 'select coalesce(jsonb_agg(jsonb_build_object(' + || '''id'', b.id, ''public'', b.public, ''file_size_limit'', b.file_size_limit, ' + || '''allowed_mime_types'', b.allowed_mime_types) order by b.id), ''[]''::jsonb) ' + || 'from storage.buckets b' + into buckets; + end if; + + -- Migration-history integrity probe. A version recorded without executed + -- statements is the fingerprint of a history repair / mark-applied row and + -- must be covered by a fail-fast guard migration + a reviewed allowlist entry + -- (docs/database-drift-detection.md, "Guard-migration contract"). + if to_regclass('supabase_migrations.schema_migrations') is not null then + if exists ( + select 1 + from pg_attribute att + where att.attrelid = 'supabase_migrations.schema_migrations'::regclass + and att.attname = 'statements' + and att.attnum > 0 + and not att.attisdropped + ) then + execute 'select coalesce(jsonb_agg(jsonb_build_object(' + || '''version'', m.version, ''name'', m.name, ' + || '''signal'', case when m.statements is null then ''null'' else ''empty'' end' + || ') order by m.version), ''[]''::jsonb) ' + || 'from supabase_migrations.schema_migrations m ' + || 'where m.statements is null or cardinality(m.statements) = 0' + into history; + history_probe := 'ok'; + else + history_probe := 'no_statements_column'; + end if; + end if; + + return snapshot || jsonb_build_object( + 'storage_buckets', buckets, + 'migration_history', history, + 'migration_history_probe', history_probe + ); +end; +$$; + +revoke execute on function public.schema_drift_snapshot() from public, anon, authenticated; +grant execute on function public.schema_drift_snapshot() to service_role; diff --git a/supabase/schema.sql b/supabase/schema.sql index 656c2f98da..4d53cfb9f0 100644 --- a/supabase/schema.sql +++ b/supabase/schema.sql @@ -3512,8 +3512,10 @@ grant execute on function public.invoke_ingestion_worker(integer) to service_rol -- Full-inventory drift snapshot backing `npm run check:drift`. The expected -- state lives in supabase/drift-manifest.json (generated from a scratch replay -- of this file via `npm run drift:manifest`). Keep this definition byte-identical --- to supabase/migrations/20260706200000_schema_drift_snapshot.sql; a unit test --- in tests/supabase-schema.test.ts enforces it. See docs/database-drift-detection.md. +-- to the latest defining migration, +-- supabase/migrations/20260818090000_schema_drift_snapshot_history_probe.sql +-- (v2: adds the migration-history integrity probe; supersedes 20260706200000); +-- tests/drift-detection.test.ts enforces the parity. See docs/database-drift-detection.md. create or replace function public.schema_drift_snapshot() returns jsonb language plpgsql @@ -3524,9 +3526,11 @@ as $$ declare snapshot jsonb; buckets jsonb := '[]'::jsonb; + history jsonb := '[]'::jsonb; + history_probe text := 'no_history_table'; begin select jsonb_build_object( - 'snapshot_version', 1, + 'snapshot_version', 2, 'captured_at', now(), 'extensions', coalesce(( select jsonb_agg(jsonb_build_object('name', e.extname, 'schema', n.nspname) order by e.extname) @@ -3637,7 +3641,37 @@ begin into buckets; end if; - return snapshot || jsonb_build_object('storage_buckets', buckets); + -- Migration-history integrity probe. A version recorded without executed + -- statements is the fingerprint of a history repair / mark-applied row and + -- must be covered by a fail-fast guard migration + a reviewed allowlist entry + -- (docs/database-drift-detection.md, "Guard-migration contract"). + if to_regclass('supabase_migrations.schema_migrations') is not null then + if exists ( + select 1 + from pg_attribute att + where att.attrelid = 'supabase_migrations.schema_migrations'::regclass + and att.attname = 'statements' + and att.attnum > 0 + and not att.attisdropped + ) then + execute 'select coalesce(jsonb_agg(jsonb_build_object(' + || '''version'', m.version, ''name'', m.name, ' + || '''signal'', case when m.statements is null then ''null'' else ''empty'' end' + || ') order by m.version), ''[]''::jsonb) ' + || 'from supabase_migrations.schema_migrations m ' + || 'where m.statements is null or cardinality(m.statements) = 0' + into history; + history_probe := 'ok'; + else + history_probe := 'no_statements_column'; + end if; + end if; + + return snapshot || jsonb_build_object( + 'storage_buckets', buckets, + 'migration_history', history, + 'migration_history_probe', history_probe + ); end; $$; diff --git a/supabase/search-health-unmonitored-indexes.json b/supabase/search-health-unmonitored-indexes.json new file mode 100644 index 0000000000..b0d9b5e1dc --- /dev/null +++ b/supabase/search-health-unmonitored-indexes.json @@ -0,0 +1,269 @@ +{ + "_comment": "Runtime index-monitoring ratchet (docs/database-drift-detection.md, 'Runtime index-monitoring ratchet'). Every repo-defined index on the retrieval-critical tables (documents, document_chunks, document_index_units, document_embedding_fields, document_memory_cards, rag_retrieval_logs) must be either monitored by search_schema_health() required_indexes/index_aliases or listed here with a reason and a disposition. `accepted-unmonitored`: absence would degrade an operational path but not clinical retrieval, and check:drift's full index inventory still reports it missing. `monitor-candidate`: retrieval-facing or currently absent on live (forensics section 1.3, 2026-08-14) — a Phase 4.4 migration extending required_indexes must decide it; it stays visibly flagged here until then. required_indexes changes travel by migration only, never by editing schema.sql. tests/search-health-index-coverage.test.ts enforces this file. Seeded 2026-08-18: 44 entries.", + "unmonitored": [ + { + "index": "documents_search_idx", + "table": "documents", + "disposition": "monitor-candidate", + "reason": "GIN over the documents search tsvector used by the title fast path; retrieval-facing but only the trgm title index is monitored today" + }, + { + "index": "documents_registry_projection_lookup_idx", + "table": "documents", + "disposition": "monitor-candidate", + "reason": "Absent on live per forensics section 1.3 (2026-08-14); registry-projection lookup path — Phase 4 restores it and Phase 4.4 decides required_indexes by migration" + }, + { + "index": "documents_owner_content_hash_unique_idx", + "table": "documents", + "disposition": "accepted-unmonitored", + "reason": "Upload dedup uniqueness for owner+content_hash; ingestion correctness, not the retrieval read path; check:drift index inventory reports absence" + }, + { + "index": "documents_owner_status_idx", + "table": "documents", + "disposition": "accepted-unmonitored", + "reason": "Owner+status listing/queue index for the documents mode and ingestion dashboards; operational, not the answer retrieval path" + }, + { + "index": "documents_import_batch_idx", + "table": "documents", + "disposition": "accepted-unmonitored", + "reason": "Bulk-import batch lookup; ingestion-side operational index with no role in retrieval RPCs" + }, + { + "index": "documents_indexing_v3_agent_claim_idx", + "table": "documents", + "disposition": "accepted-unmonitored", + "reason": "Indexing-v3 agent claim scan support; worker/queue path, not the retrieval read path" + }, + { + "index": "documents_owner_id_covering_idx", + "table": "documents", + "disposition": "accepted-unmonitored", + "reason": "Owner covering index for owner-scoped document listings; documents_indexed_owner_title_idx already covers the monitored owner+indexed retrieval predicate" + }, + { + "index": "documents_indexed_updated_at_idx", + "table": "documents", + "disposition": "accepted-unmonitored", + "reason": "Recency ordering for indexed documents in listings and registry sync; operational ordering, not retrieval matching" + }, + { + "index": "documents_owner_updated_at_indexed_idx", + "table": "documents", + "disposition": "accepted-unmonitored", + "reason": "Owner+updated_at listing index added 2026-07-20 for the documents mode; operational, not the answer retrieval path" + }, + { + "index": "documents_status_idx", + "table": "documents", + "disposition": "accepted-unmonitored", + "reason": "Plain status index declared only in schema.sql (never by a migration — drift backlog item 10, migration-chain fidelity); operational status scans" + }, + { + "index": "document_chunks_anchor_idx", + "table": "document_chunks", + "disposition": "monitor-candidate", + "reason": "Absent on live per forensics section 1.3 (2026-08-14); anchor lookup for citation deep links — Phase 4 restores it and Phase 4.4 decides required_indexes by migration" + }, + { + "index": "document_chunks_content_hash_idx", + "table": "document_chunks", + "disposition": "accepted-unmonitored", + "reason": "Chunk content-hash dedup during reindex; ingestion-side, absence slows reindex but not retrieval" + }, + { + "index": "document_chunks_generation_idx", + "table": "document_chunks", + "disposition": "accepted-unmonitored", + "reason": "Index-generation bookkeeping used by the atomic reindex commit; ingestion path, not retrieval matching" + }, + { + "index": "document_chunks_document_generation_chunk_idx", + "table": "document_chunks", + "disposition": "accepted-unmonitored", + "reason": "Unique document+generation+chunk_index guard for atomic reindex commits; ingestion correctness, not the retrieval read path" + }, + { + "index": "document_chunks_document_id_idx", + "table": "document_chunks", + "disposition": "accepted-unmonitored", + "reason": "Redundant with the monitored document_chunks_document_idx (same leading column); codified from live for FK support only" + }, + { + "index": "document_chunks_meta_rag_indexing_version_idx", + "table": "document_chunks", + "disposition": "accepted-unmonitored", + "reason": "Version-tag scan for reindex campaigns (metadata rag_indexing_version); operational, not retrieval matching" + }, + { + "index": "document_embedding_fields_dedup_idx", + "table": "document_embedding_fields", + "disposition": "accepted-unmonitored", + "reason": "Unique dedup guard on embedding fields; ingestion correctness, not the retrieval read path" + }, + { + "index": "document_embedding_fields_document_generation_idx", + "table": "document_embedding_fields", + "disposition": "accepted-unmonitored", + "reason": "Index-generation bookkeeping for atomic reindex commits; ingestion path" + }, + { + "index": "document_embedding_fields_document_id_idx", + "table": "document_embedding_fields", + "disposition": "accepted-unmonitored", + "reason": "Redundant with the monitored document_embedding_fields_document_idx (same leading column); codified from live for FK support only" + }, + { + "index": "document_embedding_fields_meta_rag_indexing_version_idx", + "table": "document_embedding_fields", + "disposition": "accepted-unmonitored", + "reason": "Schema.sql-only version-tag scan index for reindex campaigns (drift backlog item 10); operational" + }, + { + "index": "document_embedding_fields_owner_document_created_idx", + "table": "document_embedding_fields", + "disposition": "accepted-unmonitored", + "reason": "Schema.sql-only owner+document+created listing index (drift backlog item 10); the monitored document_embedding_fields_owner_chunk_idx covers the retrieval owner predicate" + }, + { + "index": "document_embedding_fields_owner_id_idx", + "table": "document_embedding_fields", + "disposition": "accepted-unmonitored", + "reason": "Schema.sql-only rename target of the dropped document_embedding_fields_owner_idx (drift backlog item 10); owner FK support, superseded for retrieval by document_embedding_fields_owner_chunk_idx" + }, + { + "index": "document_embedding_fields_search_tsv_chunk_gin_idx", + "table": "document_embedding_fields", + "disposition": "monitor-candidate", + "reason": "Schema.sql-only GIN over the embedding-field search tsvector (drift backlog item 10); lexical field retrieval path — decide required_indexes membership by migration once its live status is confirmed" + }, + { + "index": "document_embedding_fields_source_chunk_id_idx", + "table": "document_embedding_fields", + "disposition": "accepted-unmonitored", + "reason": "Schema.sql-only source_chunk_id FK support (drift backlog item 10); join support for cleanup, not retrieval matching" + }, + { + "index": "document_index_units_chunk_idx", + "table": "document_index_units", + "disposition": "accepted-unmonitored", + "reason": "chunk_id FK support for index units; join/cleanup path, the monitored owner_chunk_type index covers the retrieval predicate" + }, + { + "index": "document_index_units_document_generation_idx", + "table": "document_index_units", + "disposition": "accepted-unmonitored", + "reason": "Index-generation bookkeeping for atomic reindex commits; ingestion path" + }, + { + "index": "document_index_units_document_id_idx", + "table": "document_index_units", + "disposition": "accepted-unmonitored", + "reason": "Redundant with the monitored document_index_units_document_idx (same leading column); codified from live for FK support only" + }, + { + "index": "document_index_units_heading_path_idx", + "table": "document_index_units", + "disposition": "monitor-candidate", + "reason": "Absent on live per forensics section 1.3 (2026-08-14); heading-path scoped lookups — Phase 4 restores it and Phase 4.4 decides required_indexes by migration" + }, + { + "index": "document_index_units_image_idx", + "table": "document_index_units", + "disposition": "accepted-unmonitored", + "reason": "image_id FK support for visual index units; join/cleanup path, not the retrieval predicate" + }, + { + "index": "document_index_units_owner_document_created_idx", + "table": "document_index_units", + "disposition": "accepted-unmonitored", + "reason": "Owner+document+created listing index codified from live; operational ordering, retrieval uses owner_chunk_type" + }, + { + "index": "document_index_units_owner_id_idx", + "table": "document_index_units", + "disposition": "accepted-unmonitored", + "reason": "Plain owner FK support codified from live; the monitored document_index_units_owner_chunk_type_idx covers the retrieval owner predicate" + }, + { + "index": "document_index_units_owner_type_idx", + "table": "document_index_units", + "disposition": "accepted-unmonitored", + "reason": "Owner+unit_type prefix of the monitored owner_chunk_type index; retained for older query shapes, retrieval predicate already monitored" + }, + { + "index": "document_index_units_producer_generation_idx", + "table": "document_index_units", + "disposition": "accepted-unmonitored", + "reason": "Producer-scoped generation bookkeeping for deep-memory commits; ingestion path" + }, + { + "index": "document_index_units_search_idx", + "table": "document_index_units", + "disposition": "monitor-candidate", + "reason": "GIN over the index-unit search tsvector; lexical candidate gate for the index-unit hybrid RPC, retrieval-facing with no monitored equivalent (worst-covered table: 2 of 16 monitored)" + }, + { + "index": "document_index_units_source_chunk_id_idx", + "table": "document_index_units", + "disposition": "accepted-unmonitored", + "reason": "source_chunk_id FK support codified from live; join/cleanup path" + }, + { + "index": "document_index_units_source_image_id_idx", + "table": "document_index_units", + "disposition": "accepted-unmonitored", + "reason": "source_image_id FK support codified from live; join/cleanup path" + }, + { + "index": "document_index_units_terms_idx", + "table": "document_index_units", + "disposition": "monitor-candidate", + "reason": "GIN over normalized_terms used by index-unit lexical matching; retrieval-facing with no monitored equivalent" + }, + { + "index": "document_memory_cards_document_generation_idx", + "table": "document_memory_cards", + "disposition": "accepted-unmonitored", + "reason": "Index-generation bookkeeping for atomic reindex commits; ingestion path" + }, + { + "index": "document_memory_cards_document_id_idx", + "table": "document_memory_cards", + "disposition": "accepted-unmonitored", + "reason": "Redundant with the monitored document_memory_cards_document_idx (same leading column); codified from live for FK support only" + }, + { + "index": "document_memory_cards_owner_document_created_idx", + "table": "document_memory_cards", + "disposition": "accepted-unmonitored", + "reason": "Owner+document+created listing index codified from live; operational ordering" + }, + { + "index": "document_memory_cards_owner_idx", + "table": "document_memory_cards", + "disposition": "accepted-unmonitored", + "reason": "Owner FK/RLS support for memory cards; owner predicate is enforced in the hybrid RPC body and the monitored HNSW index carries the retrieval load" + }, + { + "index": "document_memory_cards_producer_generation_idx", + "table": "document_memory_cards", + "disposition": "accepted-unmonitored", + "reason": "Producer-scoped generation bookkeeping for deep-memory commits; ingestion path" + }, + { + "index": "document_memory_cards_search_idx", + "table": "document_memory_cards", + "disposition": "monitor-candidate", + "reason": "GIN over the memory-card search tsvector; lexical half of the memory-card hybrid RPC, retrieval-facing with no monitored equivalent" + }, + { + "index": "document_memory_cards_section_idx", + "table": "document_memory_cards", + "disposition": "accepted-unmonitored", + "reason": "section_id FK support for memory cards; join/cleanup path, not the retrieval predicate" + } + ] +} diff --git a/tests/drift-detection.test.ts b/tests/drift-detection.test.ts index 1143277ad7..b9d5c28565 100644 --- a/tests/drift-detection.test.ts +++ b/tests/drift-detection.test.ts @@ -1,7 +1,12 @@ import { readFileSync } from "node:fs"; import { join } from "node:path"; import { describe, expect, it } from "vitest"; -import { compareDriftSnapshots, normalizedSchemaSha256 } from "../scripts/check-drift"; +import { + HISTORY_PROBE_MIGRATION, + compareDriftSnapshots, + historyEntryProblems, + normalizedSchemaSha256, +} from "../scripts/check-drift"; const root = join(__dirname, ".."); const read = (relative: string) => readFileSync(join(root, relative), "utf8"); @@ -37,6 +42,15 @@ describe("drift manifest freshness (offline half of check:drift)", () => { expect((snapshot.indexes as unknown[]).length).toBeGreaterThan(100); expect((snapshot.tables as unknown[]).length).toBeGreaterThan(30); }); + + it("manifest snapshot is v2 with the migration-history probe (empty on replay — no history table)", () => { + const { snapshot } = JSON.parse(read("supabase/drift-manifest.json")); + expect(snapshot.snapshot_version).toBe(2); + // The scratch replay never creates supabase_migrations, so the manifest + // must record that honestly rather than an "ok" probe with zero rows. + expect(snapshot.migration_history_probe).toBe("no_history_table"); + expect(snapshot.migration_history).toEqual([]); + }); }); describe("local drift replay container safety", () => { @@ -65,16 +79,37 @@ describe("schema_drift_snapshot definition parity (migration vs schema.sql)", () return text.slice(start, end); }; - it("migration 20260706200000 and schema.sql carry byte-identical definitions", () => { - const fromMigration = extract(read("supabase/migrations/20260706200000_schema_drift_snapshot.sql")); + it(`migration ${HISTORY_PROBE_MIGRATION.slice(0, 14)} (latest definer) and schema.sql carry byte-identical definitions`, () => { + const fromMigration = extract(read(`supabase/migrations/${HISTORY_PROBE_MIGRATION}`)); const fromSchema = extract(read("supabase/schema.sql")); expect(fromSchema).toBe(fromMigration); }); + + it("v2 probe reads history via a guarded dynamic query and reports its own availability", () => { + const sql = extract(read(`supabase/migrations/${HISTORY_PROBE_MIGRATION}`)); + expect(sql).toContain("'snapshot_version', 2"); + expect(sql).toContain("to_regclass('supabase_migrations.schema_migrations')"); + expect(sql).toContain("m.statements is null or cardinality(m.statements) = 0"); + for (const probe of ["'no_history_table'", "'no_statements_column'", "'ok'"]) { + expect(sql, `probe state ${probe} missing`).toContain(probe); + } + expect(sql).toContain("'migration_history', history"); + expect(sql).toContain("'migration_history_probe', history_probe"); + // Still stable + security definer + pinned search_path, exactly like v1. + expect(sql).toMatch(/stable\s+security definer\s+set search_path to ''/); + }); }); describe("drift allowlist hygiene", () => { const allowlist = JSON.parse(read("supabase/drift-allowlist.json")) as { - entries: { category: string; kind: string; key: string; live_key?: string; reason: string }[]; + entries: { + category: string; + kind: string; + key: string; + live_key?: string; + reason: string; + guard?: { class: string; migration: string; objects?: string[] }; + }[]; }; it("every entry is well-formed with a real reason", () => { @@ -90,14 +125,22 @@ describe("drift allowlist hygiene", () => { "constraints", "triggers", "storage_buckets", + "migration_history", ]).toContain(entry.category); - expect(["missing_live", "unexpected_live", "mismatch", "alias"]).toContain(entry.kind); + expect(["missing_live", "unexpected_live", "mismatch", "alias", "no_statements"]).toContain(entry.kind); expect(entry.key.length).toBeGreaterThan(0); expect(entry.reason.length, `allowlist entry ${entry.category}/${entry.key} needs a reason`).toBeGreaterThan(20); expect(entry.reason).not.toContain("UNCLASSIFIED"); if (entry.kind === "alias") { expect(entry.live_key, `alias entry ${entry.key} needs live_key`).toBeTruthy(); } + // migration_history <-> no_statements is a one-to-one pairing, and every + // such entry must pass the structural guard validation check:drift uses. + if (entry.category === "migration_history" || entry.kind === "no_statements") { + expect(entry.category).toBe("migration_history"); + expect(entry.kind).toBe("no_statements"); + expect(historyEntryProblems(entry as never), `history entry ${entry.key}`).toEqual([]); + } } }); @@ -220,4 +263,81 @@ describe("compareDriftSnapshots", () => { const r2 = compareDriftSnapshots(expected, badLive, alias); expect(r2.findings.map((f: { kind: string }) => f.kind)).toContain("missing_live"); }); + + describe("migration-history probe", () => { + const withHistory = (rows: unknown[], probe = "ok") => ({ + ...clone(), + migration_history: rows, + migration_history_probe: probe, + }); + const rows = [ + { version: "20260701030000", name: "schema_health_hybrid_execution_smoke", signal: "null" }, + { version: "20260712170500", name: "codify_live_operational_indexes", signal: "empty" }, + ]; + const validEntry = { + category: "migration_history", + kind: "no_statements", + key: "20260701030000", + reason: "mark-applied after hand-apply; function redefined by the M13 guard migration", + guard: { + class: "superseded", + migration: "20260706010000_search_schema_health_m13_guard.sql", + objects: ["search_schema_health"], + }, + } as never; + + it("reports every live no-statements version as a finding, ignoring the manifest side", () => { + // The manifest replay has no history table: expected side is [] / not ok. + const expected = { ...clone(), migration_history: [], migration_history_probe: "no_history_table" }; + const r = compareDriftSnapshots(expected, withHistory(rows), []); + const history = r.findings.filter((f: { category: string }) => f.category === "migration_history"); + expect(history.map((f: { key: string }) => f.key)).toEqual(["20260701030000", "20260712170500"]); + expect(history.every((f: { kind: string }) => f.kind === "no_statements")).toBe(true); + expect(history[1].detail).toContain("codify_live_operational_indexes (statements empty)"); + // Object categories still compare cleanly; history rows never appear as unexpected_live. + expect(r.findings).toHaveLength(2); + }); + + it("a valid guard-backed allowlist entry consumes exactly its version", () => { + const r = compareDriftSnapshots(clone(), withHistory(rows), [validEntry]); + expect(r.findings.map((f: { key: string }) => f.key)).toEqual(["20260712170500"]); + expect(r.allowed).toHaveLength(1); + expect(r.staleEntries).toEqual([]); + }); + + it("a malformed or unbacked entry never silences a row and is reported stale", () => { + const noGuard = { ...(validEntry as object), guard: undefined } as never; + const missingFile = { + ...(validEntry as object), + guard: { class: "superseded", migration: "20260706010001_does_not_exist.sql", objects: ["x"] }, + } as never; + const wrongClassForNewVersion = { + ...(validEntry as object), + key: "20260901000000", + guard: { class: "superseded", migration: "20260901010000_later.sql", objects: ["x"] }, + } as never; + const laterRows = [...rows, { version: "20260901000000", name: "future_repair", signal: "null" }]; + const r = compareDriftSnapshots(clone(), withHistory(laterRows), [noGuard, missingFile, wrongClassForNewVersion]); + expect(r.findings.map((f: { key: string }) => f.key)).toEqual([ + "20260701030000", + "20260712170500", + "20260901000000", + ]); + expect(r.staleEntries).toHaveLength(3); + expect(historyEntryProblems(noGuard)).toContain("guard is required"); + expect(historyEntryProblems(missingFile).join(" ")).toContain("does not exist"); + expect(historyEntryProblems(wrongClassForNewVersion, { migrationExists: () => true }).join(" ")).toContain( + "must use a validation guard", + ); + }); + + it("explains a live snapshot without the probe as a pending deploy, not as drift", () => { + const r = compareDriftSnapshots(clone(), clone(), []); + expect(r.findings).toEqual([]); + expect(r.infos.some((i: string) => i.includes(HISTORY_PROBE_MIGRATION))).toBe(true); + const r2 = compareDriftSnapshots(clone(), withHistory([], "no_statements_column"), []); + expect(r2.findings).toEqual([]); + expect(r2.infos.some((i: string) => i.includes("no_statements_column"))).toBe(true); + }); + }); }); diff --git a/tests/migration-history-guards.test.ts b/tests/migration-history-guards.test.ts new file mode 100644 index 0000000000..f26c6779a6 --- /dev/null +++ b/tests/migration-history-guards.test.ts @@ -0,0 +1,155 @@ +import { existsSync, readFileSync, readdirSync } from "node:fs"; +import { join } from "node:path"; +import { describe, expect, it } from "vitest"; +import { + GUARD_CONTRACT_VERSION, + HISTORY_GUARD_CLASSES, + HISTORY_PROBE_MIGRATION, + historyEntryProblems, + type AllowlistEntry, +} from "../scripts/check-drift"; + +/** + * Guard-migration contract (docs/database-drift-detection.md, plan phase 6.2). + * + * Every `migration_history` allowlist entry — a supabase_migrations version the + * live probe reports as recorded WITHOUT executed statements — must point at a + * guard migration that really covers it: + * validation a later fail-fast validation migration (20260804110240 pattern) + * that raises when the listed objects are missing/invalid; + * superseded a later migration that re-created every listed object + * (pre-contract history only); + * no_ddl the version's own file, which has no effect at all. + * check:drift enforces the structural half at runtime (an invalid entry can + * never silence a row); this test enforces the object-level half offline so a + * pointer to the wrong file cannot pass review as a "guard". + */ + +const root = join(__dirname, ".."); +const migrationsDir = join(root, "supabase", "migrations"); +const read = (relative: string) => readFileSync(join(root, relative), "utf8"); +const readMigration = (fileName: string) => readFileSync(join(migrationsDir, fileName), "utf8"); + +const allowlist = JSON.parse(read("supabase/drift-allowlist.json")) as { entries: AllowlistEntry[] }; +const historyEntries = allowlist.entries.filter((entry) => entry.category === "migration_history"); +const migrationFiles = readdirSync(migrationsDir).filter((name) => /^\d{14}_.+\.sql$/.test(name)); + +const escape = (name: string) => name.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"); + +/** Does `sql` create/replace `object` (function, index, table, view, policy, trigger)? */ +function createsObject(sql: string, object: string): boolean { + const name = escape(object); + const patterns = [ + new RegExp(`create\\s+or\\s+replace\\s+function\\s+(?:public\\.)?${name}\\s*\\(`, "i"), + new RegExp(`create\\s+function\\s+(?:public\\.)?${name}\\s*\\(`, "i"), + new RegExp( + `create\\s+(?:unique\\s+)?index\\s+(?:concurrently\\s+)?(?:if\\s+not\\s+exists\\s+)?${name}\\s+on\\b`, + "i", + ), + new RegExp(`create\\s+table\\s+(?:if\\s+not\\s+exists\\s+)?(?:public\\.)?${name}\\s*\\(`, "i"), + new RegExp(`create\\s+(?:or\\s+replace\\s+)?(?:materialized\\s+)?view\\s+(?:public\\.)?${name}\\b`, "i"), + new RegExp(`create\\s+policy\\s+"?${name}"?\\s+on\\b`, "i"), + new RegExp(`create\\s+(?:or\\s+replace\\s+)?trigger\\s+${name}\\b`, "i"), + ]; + return patterns.some((pattern) => pattern.test(sql)); +} + +/** SQL with block/line comments removed and whitespace collapsed. */ +function stripSql(sql: string): string { + return sql + .replace(/\/\*[\s\S]*?\*\//g, "") + .replace(/--[^\n]*/g, "") + .replace(/\s+/g, " ") + .trim(); +} + +describe("migration-history probe and guard-migration contract", () => { + it("the v2 snapshot migration exists and check:drift knows its name", () => { + expect(existsSync(join(migrationsDir, HISTORY_PROBE_MIGRATION))).toBe(true); + expect(read("scripts/check-drift.ts")).toContain("migration_history"); + expect(read("scripts/check-drift.ts")).toContain("no_statements"); + }); + + it("the contract is written down where operators and agents will read it", () => { + const doc = read("docs/database-drift-detection.md"); + expect(doc).toContain("## Guard-migration contract"); + expect(doc).toContain("## Migration-history probe"); + expect(doc).toContain("## Runtime index-monitoring ratchet"); + const agents = read("AGENTS.md"); + expect(agents).toMatch(/fail-fast\s+validation\s+(guard\s+)?migration/i); + expect(agents).toContain("20260804110240"); + expect(agents).toContain("migration_history"); + }); + + it("the guard classes are exactly the documented set and the contract date is fixed", () => { + expect([...HISTORY_GUARD_CLASSES]).toEqual(["validation", "superseded", "no_ddl"]); + expect(GUARD_CONTRACT_VERSION).toBe("20260818000000"); + }); + + it("every allowlisted no-statements version passes the structural guard validation", () => { + for (const entry of historyEntries) { + expect(historyEntryProblems(entry), `allowlist entry ${entry.key}`).toEqual([]); + } + }); + + it("every allowlisted no-statements version has a matching guard migration file that really covers it", () => { + for (const entry of historyEntries) { + const guard = entry.guard!; + const label = `allowlist entry ${entry.key} (${guard.class} → ${guard.migration})`; + expect(migrationFiles, `${label}: guard file missing`).toContain(guard.migration); + const guardSql = readMigration(guard.migration); + + if (guard.class === "validation") { + // A validation guard never builds: it raises when the objects are + // missing/invalid/mismatched (the 20260804110240 pattern). + expect(guardSql, `${label}: validation guard must raise`).toMatch(/raise\s+exception/i); + expect(guardSql, `${label}: validation guard must not create the objects it validates`).not.toMatch( + /create\s+(?:unique\s+)?index\s+(?:concurrently\s+)?(?:if\s+not\s+exists\s+)?[a-z_]/i, + ); + for (const object of guard.objects ?? []) { + expect(guardSql, `${label}: guard does not mention object ${object}`).toContain(object); + } + } else if (guard.class === "superseded") { + expect(guard.objects?.length, `${label}: superseded guard must list the objects it re-creates`).toBeGreaterThan( + 0, + ); + for (const object of guard.objects ?? []) { + expect(createsObject(guardSql, object), `${label}: guard does not re-create ${object}`).toBe(true); + } + // The version's own file must define at least the objects claimed — + // otherwise the entry is describing the wrong version. + const own = migrationFiles.find((name) => name.startsWith(`${entry.key}_`)); + expect(own, `${label}: the allowlisted version has no local migration file`).toBeTruthy(); + for (const object of guard.objects ?? []) { + expect(createsObject(readMigration(own!), object), `${label}: ${own} does not define ${object}`).toBe(true); + } + } else if (guard.class === "no_ddl") { + const body = stripSql(guardSql); + expect( + body === "" || /^select\s+1\s*;?$/i.test(body), + `${label}: no_ddl guard has effective SQL: ${body.slice(0, 80)}`, + ).toBe(true); + } + } + }); + + it("no pre-contract class is used for a version at or after the contract date", () => { + for (const entry of historyEntries) { + if (entry.key >= GUARD_CONTRACT_VERSION) { + expect(entry.guard?.class, `allowlist entry ${entry.key} must use a validation guard`).toBe("validation"); + } + } + }); + + it("the seeded entries resolve to the section 1.1 mark-applied cluster by name", () => { + // Guards against a typo in the version key: each seeded version must have a + // local file whose stem is one of the names the forensics file recorded. + const forensics = read("docs/audit/live-drift-forensics-2026-08.md"); + for (const entry of historyEntries) { + const own = migrationFiles.find((name) => name.startsWith(`${entry.key}_`)); + expect(own, `allowlist entry ${entry.key} has no local migration`).toBeTruthy(); + const stem = own!.replace(/^\d{14}_/, "").replace(/\.sql$/, ""); + expect(forensics, `${own} is not a name recorded in forensics §1.1`).toContain(`\`${stem}\``); + } + }); +}); diff --git a/tests/search-health-index-coverage.test.ts b/tests/search-health-index-coverage.test.ts new file mode 100644 index 0000000000..3056ff46a0 --- /dev/null +++ b/tests/search-health-index-coverage.test.ts @@ -0,0 +1,202 @@ +import { readFileSync, readdirSync } from "node:fs"; +import { join } from "node:path"; +import { describe, expect, it } from "vitest"; + +/** + * Runtime index-monitoring ratchet (docs/database-drift-detection.md, plan + * phase 6.3). `search_schema_health()` monitors a curated `required_indexes` + * list; the 20 repo-defined indexes found absent on live in 2026-08 were + * invisible to it. This test makes every monitoring decision on the + * retrieval-critical tables explicit: each repo-defined index there is either + * monitored (required_indexes / index_aliases) or listed with a reason and a + * disposition in supabase/search-health-unmonitored-indexes.json. + * + * "Repo-defined" is computed two ways and unioned, so neither source can hide + * an index from the other: (a) an order-aware replay of every create/drop + * index statement in supabase/migrations, (b) supabase/drift-manifest.json + * (schema.sql replayed from scratch). Constraint-backed primary-key indexes + * are exempt: the constraint inventory in check:drift already guards them. + */ + +const root = join(__dirname, ".."); +const migrationsDir = join(root, "supabase", "migrations"); +const read = (relative: string) => readFileSync(join(root, relative), "utf8"); + +export const RETRIEVAL_CRITICAL_TABLES = [ + "documents", + "document_chunks", + "document_index_units", + "document_embedding_fields", + "document_memory_cards", + "rag_retrieval_logs", +] as const; + +type Disposition = "accepted-unmonitored" | "monitor-candidate"; +type UnmonitoredEntry = { index: string; table: string; reason: string; disposition: Disposition }; +type CoverageFile = { _comment?: string; unmonitored: UnmonitoredEntry[] }; + +const stripSql = (sql: string) => sql.replace(/\/\*[\s\S]*?\*\//g, "").replace(/--[^\n]*/g, ""); + +/** Order-aware replay of create/drop index across the migration chain. */ +function migrationIndexes(): Map { + const live = new Map(); // index -> table + const files = readdirSync(migrationsDir) + .filter((name) => /^\d{14}_.+\.sql$/.test(name)) + .sort(); + const statement = + /\b(create\s+(?:unique\s+)?index\s+(?:concurrently\s+)?(?:if\s+not\s+exists\s+)?([a-z0-9_]+)\s+on\s+(?:only\s+)?(?:public\.)?([a-z0-9_]+)|drop\s+index\s+(?:concurrently\s+)?(?:if\s+exists\s+)?([^;]+?))\s*(?=[;(]|\busing\b|\bcascade\b|\brestrict\b)/gi; + for (const file of files) { + const sql = stripSql(readFileSync(join(migrationsDir, file), "utf8")); + for (const match of sql.matchAll(statement)) { + if (match[2]) { + live.set(match[2].toLowerCase(), match[3].toLowerCase()); + } else if (match[4]) { + for (const raw of match[4].split(",")) { + const name = raw + .trim() + .replace(/^public\./i, "") + .toLowerCase(); + if (name) live.delete(name); + } + } + } + } + return live; +} + +function manifestIndexes(): { indexes: Map; primaryKeys: Set } { + const { snapshot } = JSON.parse(read("supabase/drift-manifest.json")) as { + snapshot: { + indexes: { name: string; table: string; def: string }[]; + constraints: { name: string; table: string; def: string }[]; + }; + }; + const indexes = new Map(); + for (const row of snapshot.indexes) indexes.set(row.name, row.table); + const primaryKeys = new Set(); + for (const con of snapshot.constraints) { + if (/^PRIMARY KEY/i.test(con.def)) primaryKeys.add(con.name); + } + return { indexes, primaryKeys }; +} + +/** required_indexes + every index_aliases value from the LATEST search_schema_health definer. */ +function monitoredIndexes(): { monitored: Set; definer: string } { + const files = readdirSync(migrationsDir) + .filter((name) => /^\d{14}_.+\.sql$/.test(name)) + .sort(); + const definerPattern = + /create\s+or\s+replace\s+function\s+public\.search_schema_health\s*\(\)[\s\S]*?\$\$[\s\S]*?\$\$/i; + let definer = ""; + let body = ""; + for (const file of files) { + const sql = readFileSync(join(migrationsDir, file), "utf8"); + const match = sql.match(definerPattern); + if (match) { + definer = file; + body = match[0]; + } + } + expect(definer, "no migration defines search_schema_health()").not.toBe(""); + const parse = (text: string) => { + const required = text.match(/required_indexes\s+constant\s+text\[\]\s*:=\s*array\[([\s\S]*?)\]/i); + expect(required, `required_indexes not found in ${definer}`).toBeTruthy(); + const names = [...required![1].matchAll(/'([a-z0-9_]+)'/g)].map((m) => m[1]); + const aliases = text.match(/index_aliases\s+constant\s+jsonb\s*:=\s*jsonb_build_object\(([\s\S]*?)\);/i); + expect(aliases, `index_aliases not found in ${definer}`).toBeTruthy(); + const aliasNames = [...aliases![1].matchAll(/'([a-z0-9_]+)'/g)].map((m) => m[1]); + return new Set([...names, ...aliasNames]); + }; + const fromMigration = parse(body); + // schema.sql must carry the same monitored set — required_indexes changes + // travel by migration only, never by editing the mirror. + const schemaMatch = read("supabase/schema.sql").match(definerPattern); + expect(schemaMatch, "schema.sql does not define search_schema_health()").toBeTruthy(); + const fromSchema = parse(schemaMatch![0]); + expect([...fromSchema].sort()).toEqual([...fromMigration].sort()); + return { monitored: fromMigration, definer }; +} + +function loadCoverage(): CoverageFile { + return JSON.parse(read("supabase/search-health-unmonitored-indexes.json")) as CoverageFile; +} + +describe("search_schema_health() index-monitoring ratchet on retrieval-critical tables", () => { + const scope = new Set(RETRIEVAL_CRITICAL_TABLES); + const fromMigrations = migrationIndexes(); + const { indexes: fromManifest, primaryKeys } = manifestIndexes(); + const { monitored, definer } = monitoredIndexes(); + const coverage = loadCoverage(); + + const candidates = new Map(); + for (const [name, table] of fromMigrations) + if (scope.has(table) && !primaryKeys.has(name)) candidates.set(name, table); + for (const [name, table] of fromManifest) if (scope.has(table) && !primaryKeys.has(name)) candidates.set(name, table); + + it("parses a plausible inventory (sanity floors)", () => { + expect(definer).toMatch(/^\d{14}_.+\.sql$/); + expect(monitored.size).toBeGreaterThanOrEqual(22); + expect(candidates.size).toBeGreaterThan(40); + // The manifest and the migration replay largely agree; a wild disagreement + // means the replay parser or the manifest is broken, not the schema. + const onlyInMigrations = [...fromMigrations].filter(([n, t]) => scope.has(t) && !fromManifest.has(n)); + const onlyInManifest = [...fromManifest].filter( + ([n, t]) => scope.has(t) && !fromMigrations.has(n) && !primaryKeys.has(n), + ); + expect( + onlyInMigrations.length + onlyInManifest.length, + `migration-vs-manifest disagreement: ${JSON.stringify({ onlyInMigrations, onlyInManifest })}`, + ).toBeLessThan(12); + }); + + it("every repo-defined index on a retrieval-critical table is monitored or explicitly listed as unmonitored", () => { + const listed = new Map(coverage.unmonitored.map((entry) => [entry.index, entry])); + const uncovered = [...candidates] + .filter(([name]) => !monitored.has(name) && !listed.has(name)) + .map(([name, table]) => `${table}.${name}`) + .sort(); + expect( + uncovered, + `${uncovered.length} index(es) on retrieval-critical tables are neither in search_schema_health() required_indexes/index_aliases (${definer}) nor listed in supabase/search-health-unmonitored-indexes.json:\n ${uncovered.join("\n ")}`, + ).toEqual([]); + }); + + it("every unmonitored entry is well-formed, current, and not secretly monitored", () => { + const seen = new Set(); + for (const entry of coverage.unmonitored) { + const label = `unmonitored entry ${entry.table}.${entry.index}`; + expect(seen.has(entry.index), `${label} is duplicated`).toBe(false); + seen.add(entry.index); + expect(scope.has(entry.table), `${label}: table is outside the retrieval-critical scope`).toBe(true); + expect(candidates.get(entry.index), `${label}: index is not defined by the repo (stale entry — remove it)`).toBe( + entry.table, + ); + expect( + monitored.has(entry.index), + `${label}: index IS monitored by search_schema_health — remove the stale entry`, + ).toBe(false); + expect(entry.reason?.trim().length ?? 0, `${label}: reason must be a real explanation`).toBeGreaterThan(20); + expect(["accepted-unmonitored", "monitor-candidate"], `${label}: disposition`).toContain(entry.disposition); + } + }); + + it("names the section 1.3 absent indexes that fall inside this scope as monitor candidates", () => { + // Forensics §1.3 (2026-08-14): of the 20 repo-defined indexes absent on live, + // exactly these three sit on retrieval-critical tables. Until Phase 4 restores + // them and Phase 4.4 decides required_indexes by migration, they must stay + // visibly flagged rather than quietly accepted. + const absentInScope = [ + "document_chunks_anchor_idx", + "document_index_units_heading_path_idx", + "documents_registry_projection_lookup_idx", + ]; + for (const name of absentInScope) { + const entry = coverage.unmonitored.find((candidate) => candidate.index === name); + const isMonitored = monitored.has(name); + expect( + isMonitored || entry?.disposition === "monitor-candidate", + `${name} must be monitored or a monitor-candidate`, + ).toBe(true); + } + }); +});