feat(db): migration-history probe, guard-migration contract, and index-monitoring ratchet (remediation Phase 6)

AGENTS.md

@@ -415,6 +415,19 @@ Output-style plugins such as caveman mode may compress prose. They must never co
 - Bare-image storage scaffolding must discover its local schema owner at runtime and must never be reused as hosted migration SQL.
 - Run `npm run check:migration-role` after changing Supabase SQL, migration tooling, CI replay, or disaster-recovery instructions.
 - Run `npm run check:supabase-project` after changing Supabase env values.
+- **Guard-migration contract.** Any mark-applied version, `supabase migration repair --status applied`,
+  hand-applied SQL later recorded as a migration, or other history repair MUST ship a fail-fast
+  validation guard migration in the same change, following `20260804110240_restore_rag_search_health_indexes.sql`
+  exactly (validates presence + `indisvalid`/`indisready` + normalized definition, never builds,
+  `set local` timeouts, one `raise exception`). `schema_drift_snapshot()` v2 (`20260818090000`) reports every
+  `supabase_migrations` version recorded without executed statements; `check:drift` fails on any such row
+  that lacks a reviewed `migration_history` entry in `supabase/drift-allowlist.json` pointing at its guard
+  (`guard.class` `validation` is mandatory for versions from 2026-08-18; `superseded`/`no_ddl` are for
+  pre-contract history only). Never allowlist a history row bare, and never widen an entry's class to make
+  it pass. Enforced offline by `tests/migration-history-guards.test.ts`; index-monitoring decisions on the
+  retrieval-critical tables are enforced by `tests/search-health-index-coverage.test.ts` +
+  `supabase/search-health-unmonitored-indexes.json` (`required_indexes` changes travel by migration only).
+  Full contract: `docs/database-drift-detection.md`.
 
 <!-- END:supabase-project-safety -->
 

docs/audit/live-drift-forensics-2026-08.md

@@ -274,3 +274,32 @@ The A1/S1 packet must re-verify `generation_quality_gate:*` dominance on healthy
 choosing any code mitigation. Residual: hybrid fan-out still costs ~8.5 s worst-observed — owned
 by the remaining remediation phases, not a route-budget change (`#231`'s stop condition stands).
 `check:production-readiness` on the final state is **pending**.
+
+## Phase 6 — Future-proofing (repo-side; one migration authored, NOT deployed)
+
+_2026-08-18 (repo-only session; no hosted read or mutation)._ Built per plan §6.1–6.3, worker chat
+without a ledger row (residual queued via `npm run issues:add`, not on `#316`):
+
+- **6.1 History-integrity probe.** `20260818090000_schema_drift_snapshot_history_probe.sql` redefines
+  `public.schema_drift_snapshot()` (v2) to also return `migration_history` — every
+  `supabase_migrations.schema_migrations` version with `statements IS NULL` or empty — plus
+  `migration_history_probe` (`ok` / `no_history_table` / `no_statements_column`). Mirrored into
+  `schema.sql`; `drift-manifest.json` regenerated (Docker replay executed the new body: probe
+  `no_history_table`, `snapshot_version` 2). `check:drift` reports each live row as
+  `! [migration_history] no_statements <version>` unless a validated allowlist entry covers it.
+  **Not deployed** — needs the owner-approved production migration window (approval map, Phase 6.1,
+  after Phase 4). Until then the live run shows the `schema_drift_snapshot()` function `def_hash`
+  mismatch (repo-ahead) and an info line naming the pending deploy.
+- **6.2 Guard-migration contract.** Written into `docs/database-drift-detection.md` and `AGENTS.md`
+  ("Supabase project safety"). Allowlist `migration_history` entries carry `guard {class, migration,
+objects}`; classes `validation` (mandatory from 2026-08-18), `superseded`, `no_ddl`.
+  `tests/migration-history-guards.test.ts` verifies each guard file really covers its objects. Seeded
+  five §1.1 versions with repo-provable `superseded` guards (`20260701010000`, `20260701020000`,
+  `20260701030000`, `20260701060000`, `20260702000000`); the remaining §1.1 rows and the 2026-07-12
+  batch are deliberately **not** allowlisted and are the expected findings of the first post-deploy run.
+- **6.3 Runtime coverage ratchet.** `tests/search-health-index-coverage.test.ts` +
+  `supabase/search-health-unmonitored-indexes.json`: every repo-defined index on the six
+  retrieval-critical tables is monitored or listed with reason + disposition. Failed with exactly 44
+  names before the list existed; passes with 44 entries (8 `monitor-candidate`, including the three
+  §1.3-absent indexes on those tables: `document_chunks_anchor_idx`,
+  `document_index_units_heading_path_idx`, `documents_registry_projection_lookup_idx`).

docs/branch-review-records/014fee0f5a9c0fc049bfa0047afa855ccd498160d291ebb5c3d2eb0b0749a7cf.record.md

@@ -0,0 +1 @@
+| 2026-08-17 | claude/database-drift-allowlist-48839e | ccbb4cd8a95fd074f70931f73e79b31b525ef579 | remediation Phase 6: schema_drift_snapshot v2 history probe migration (not deployed), check-drift migration_history allowlist + guard contract, search-health index-monitoring ratchet, PR #2058 | self-review: migration authored + mirrored + manifest regenerated, not deployed; five superseded guards seeded, rest left as expected findings; 44-entry ratchet list red-then-green | check:migration-role pass; drift:manifest replay 46s; focused schema/drift suite 8 files 127 passed; verify:pr-local all gates green except tests/session-start-hook.test.ts (env: bash head 127 / temp EPERM, unrelated); check:rag:fixtures 36 golden cases pass |

docs/database-drift-detection.md

@@ -1,6 +1,6 @@
 # Database drift detection (`npm run check:drift`)
 
-Last updated: 2026-07-10
+Last updated: 2026-08-18 (migration-history probe, guard-migration contract, index-monitoring ratchet — remediation plan Phase 6)
 
 This repo's worst operational incidents were live-vs-repo schema drift: hybrid
 retrieval RPCs silently broken on live for an unknown period, and migrations
@@ -13,11 +13,12 @@ application-owned object against `supabase/schema.sql`.
 
 Three committed artifacts:
 
-| Artifact                                                       | Role                                                                                                                                                                                                                                        |
-| -------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `supabase/migrations/20260706200000_schema_drift_snapshot.sql` | `public.schema_drift_snapshot()` — service-role-only RPC returning the normalized live inventory (also declared in `supabase/schema.sql`; a test enforces byte parity).                                                                     |
-| `supabase/drift-manifest.json`                                 | The expected state: the same snapshot captured from a **from-scratch replay of `supabase/schema.sql`** into a disposable `supabase/postgres` Docker container (`npm run drift:manifest`). Embeds the sha256 of the schema.sql it came from. |
-| `supabase/drift-allowlist.json`                                | Known, documented divergence (each entry has a `reason`). Reported as warnings; anything not listed fails the check.                                                                                                                        |
+| Artifact                                                                     | Role                                                                                                                                                                                                                                                             |
+| ---------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `supabase/migrations/20260818090000_schema_drift_snapshot_history_probe.sql` | `public.schema_drift_snapshot()` v2 — service-role-only RPC returning the normalized live inventory plus the migration-history probe (supersedes `20260706200000`; also declared in `supabase/schema.sql`; a test enforces byte parity with the latest definer). |
+| `supabase/drift-manifest.json`                                               | The expected state: the same snapshot captured from a **from-scratch replay of `supabase/schema.sql`** into a disposable `supabase/postgres` Docker container (`npm run drift:manifest`). Embeds the sha256 of the schema.sql it came from.                      |
+| `supabase/drift-allowlist.json`                                              | Known, documented divergence (each entry has a `reason`; `migration_history` entries also need a `guard`). Reported as warnings; anything not listed fails the check.                                                                                            |
+| `supabase/search-health-unmonitored-indexes.json`                            | The runtime index-monitoring ratchet: every repo-defined index on a retrieval-critical table that `search_schema_health()` does not monitor, with a reason and disposition (see below).                                                                          |
 
 `npm run check:drift` (needs live service-role env) verifies the project ref,
 fails fast if the manifest is stale, calls the RPC, diffs, applies the
@@ -78,6 +79,143 @@ Both are decisions rather than defects: adding validity to the snapshot RPC is
 a migration, and raising the cadence spends provider budget. Recorded so the
 gap is chosen, not assumed away.
 
+A third limit was closed on the repo side by remediation-plan Phase 6 (2026-08-18)
+and is live once migration `20260818090000` is deployed: **history repairs were
+invisible.** The check compared object state only, so a `supabase_migrations`
+version recorded without executed DDL stayed silent until its objects went
+missing. The probe below turns that into a finding.
+
+## Migration-history probe
+
+`schema_drift_snapshot()` v2 (migration
+`20260818090000_schema_drift_snapshot_history_probe.sql`; plan §6.1;
+evidence `docs/audit/live-drift-forensics-2026-08.md` §1.1/§1.3) adds two keys:
+
+| Key                       | Value                                                                                                                                                                                           |
+| ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `migration_history`       | `[{version, name, signal}]` — every `supabase_migrations.schema_migrations` row where `statements IS NULL` (`signal: "null"`) or `cardinality(statements) = 0` (`"empty"`), ordered by version. |
+| `migration_history_probe` | `"ok"`, `"no_history_table"` (the schema does not exist — true of every `drift:manifest` replay container), or `"no_statements_column"` (very old CLI history table). Never a silent `[]`.      |
+
+Why the fingerprint matters: the CLI records the executed statements on every
+`db push`; a row with none is a mark-applied / `migration repair --status
+applied` / hand-applied version whose DDL the CLI never ran. §1.1 found the
+2026-07-01…02 cluster and the 2026-07-12 batch in that state. §1.3 also found
+migrations that **did** record executed DDL yet whose indexes are absent, so
+the probe is a second signal beside the object inventory, never a replacement.
+
+How `check:drift` treats it (`scripts/check-drift.ts`):
+
+- The category is **never compared manifest-vs-live** — the manifest holds
+  `migration_history_probe: "no_history_table"` and `[]`, and the generic
+  category loop deliberately excludes it. Each live row is a finding of kind
+  `no_statements` unless a validated `migration_history` allowlist entry covers
+  that exact version. Output line: `  ! [migration_history] no_statements
+<version> :: <name> (statements null|empty) — …`, captured by
+  `live-drift.yml`'s findings grep and routed into the pinned issue.
+- A malformed entry (missing/unknown class, guard file absent, wrong ordering,
+  pre-contract class on a post-contract version) never matches: the row stays
+  a finding and the entry is printed under stale entries with the reasons.
+- If the live payload lacks the key (probe not deployed), the run prints an
+  `info:` line and the function `def_hash` mismatch on `schema_drift_snapshot`
+  itself is the visible "deploy pending" signal — that is the ordinary
+  repo-ahead mechanism, not a special case. Deployment is a separately approved
+  production migration window (plan approval map, Phase 6.1, after Phase 4).
+
+**Expected first live run after deployment.** Only five versions are seeded in
+the allowlist (below); the remaining §1.1 rows — `20260701040000
+drop_dead_drifted_hybrid_variants`, `20260702100000
+add_claim_ingestion_jobs_comment`, `20260702110000 drop_redundant_indexes`,
+`20260702120000 rag_retrieval_logs_retention`, `20260702130000
+storage_cleanup_jobs_document_fk`, `20260702140000
+fix_reset_document_index_duplicate`, `20260702150000
+documents_owner_covering_index`, `20260702160000 fix_invoke_agent_url_to_guc`,
+`20260702180000 promote_index_generation_id_columns`, and the 2026-07-12 batch
+`20260712165915`…`20260712173000` — have **no repo-provable guard** and will be
+reported as findings. That is the intended behaviour ("a history-repair row
+without a validating guard migration becomes permanent, visible drift"); the
+follow-up is to author fail-fast guard migrations for them (Phase 4.4 batches
+cover the index ones) and allowlist each with a `validation` guard, not to
+allowlist them bare. Versions the live probe does not report surface as stale
+entries, which is how a wrong seed is caught.
+
+## Guard-migration contract
+
+**Rule (also in `AGENTS.md`, "Supabase project safety"): any mark-applied
+version, `supabase migration repair --status applied`, hand-applied SQL that
+is later recorded as a migration, or other history repair MUST ship a
+fail-fast validation migration in the same change, following
+`supabase/migrations/20260804110240_restore_rag_search_health_indexes.sql`
+exactly.** Such a guard:
+
+- **validates, never builds** — no `create index`, no `create or replace` of
+  the objects it guards; it checks presence (`to_regclass` /
+  `to_regprocedure`), `pg_index.indisvalid AND indisready` for indexes, and a
+  normalized `pg_get_indexdef` / `pg_get_functiondef` match against the pinned
+  canonical definition;
+- uses `set local lock_timeout` / `set local statement_timeout` (never bare
+  `set`), and raises one `raise exception … Missing: %; Invalid: %;
+Mismatched: %` naming every failure;
+- is marked applied only after the live validation passes, and its file name
+  is what the allowlist entry points at.
+
+Allowlist entry shape (`supabase/drift-allowlist.json`):
+
+```json
+{
+  "category": "migration_history",
+  "kind": "no_statements",
+  "key": "<14-digit version>",
+  "reason": "why the row has no executed DDL (mark-applied after prebuild, repair, rename …) — > 20 chars",
+  "guard": { "class": "validation", "migration": "<later>_<stem>.sql", "objects": ["<index or function name>", "…"] }
+}
+```
+
+| `guard.class` | Meaning                                                                                                                                                    | Machine check (`check:drift` structural + `tests/migration-history-guards.test.ts` object-level)                                                 |
+| ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
+| `validation`  | A later fail-fast guard migration proves the listed objects. **Required for every version ≥ `20260818000000`.**                                            | file exists, version > key, contains `raise exception`, creates no index, mentions every `objects` name; `objects` non-empty                     |
+| `superseded`  | A later migration re-created every listed object with recorded statements (squashed baseline, renumber, hotfix later codified). Pre-contract history only. | file exists, version > key, `create … <object>` for every listed object in the guard file **and** in the version's own file; key < contract date |
+| `no_ddl`      | The version's own file has no effect (comments only / `select 1;` placeholder).                                                                            | `guard.migration` is the version's own file; stripped body is empty or `select 1;`                                                               |
+
+Retiring an entry: when a version is genuinely re-recorded with statements (or
+history is squashed and the row disappears) the entry shows as stale on the
+next run — delete it. Never widen a class or drop `objects` to make an entry
+pass; the finding is the point.
+
+## Runtime index-monitoring ratchet
+
+`search_schema_health()` monitors a curated `required_indexes` list (22 names
+in the latest definer, `20260706010000_search_schema_health_m13_guard.sql`)
+plus `index_aliases`; the 20 indexes absent on live in 2026-08 were invisible
+to it. `tests/search-health-index-coverage.test.ts` now requires that **every
+repo-defined index on the retrieval-critical tables** — `documents`,
+`document_chunks`, `document_index_units`, `document_embedding_fields`,
+`document_memory_cards`, `rag_retrieval_logs` — is either monitored (in
+`required_indexes` or an alias value) or listed in
+`supabase/search-health-unmonitored-indexes.json` with a `reason` (> 20 chars)
+and a `disposition`:
+
+- `accepted-unmonitored` — absence would degrade an operational path
+  (ingestion bookkeeping, FK support, listings) but not clinical retrieval;
+  `check:drift`'s full index inventory still reports it missing.
+- `monitor-candidate` — retrieval-facing (`*_search_idx` / `*_terms_idx` GINs)
+  or currently absent on live per forensics §1.3 (`document_chunks_anchor_idx`,
+  `document_index_units_heading_path_idx`,
+  `documents_registry_projection_lookup_idx` — the three of the 20 that sit on
+  these tables; the other 17 are on tables outside this scope). A Phase 4.4
+  migration extending `required_indexes` must decide each; they stay flagged
+  until then.
+
+"Repo-defined" is computed two ways and unioned — an order-aware replay of
+every `create/drop index` in `supabase/migrations/`, and the manifest's
+`snapshot.indexes` — so a schema.sql-only index (drift backlog item 10) or a
+migration-only one cannot hide. Constraint-backed `*_pkey` indexes are exempt
+(the constraint inventory guards them). The list is also checked for stale
+entries (index no longer defined, or now monitored) and duplicates.
+`required_indexes` changes travel by migration only, never by editing
+`schema.sql`; the test cross-checks the two copies. Seeded 2026-08-18 with 44
+entries (8 monitor candidates); the test failed with exactly those 44 names
+before the list existed.
+
 ### Workflow
 
 - Change `supabase/schema.sql` → run `npm run drift:manifest` (Docker) in the