Replay the migration chain onto staging to parity, and record that the chain does not reproduce schema.sql (Phase 2, #056)

docs/branch-review-records/d54d14f00e65448e6ced1f60805bc47334ea1e7147a46a5f13b0b8e4e7ef818e.record.md

@@ -0,0 +1 @@
+| 2026-08-18 | claude/db-remediation-phase-2-a0c20b | 523f8a55b36f7ef1a7b2d149d3030cb768f95dc6 | Phase 2 staging parity replay (#056): 28-migration chain replayed onto Clinical KB Staging via authorized MCP connector with md5 byte-verification; scripts/check-drift.ts staging-key fix; check:drift run against staging (red, 19 findings); forensics Phase 2 section; two #056 inbox requests (one cancel, one update) | Self-review passed. Replay proven: 194/194 parity, zero statements IS NULL, all 28 rows md5-identical to repo files, staging corpus untouched (0 documents). Drift check red with 19 findings, correctly interpreted as chain-vs-schema.sql divergence rather than staging staleness; nothing patched. No migration, schema.sql or drift-manifest changed. Production never a mutation target. Known scope limit: measured at base ed43a64f2; origin/main has since advanced to 195 migrations including a schema_drift_snapshot history probe, so a re-measure is owed and is stated in the PR body. | verify:pr-local (green through lint+typecheck); npm run test 2 pre-existing unrelated failures with disjoint run-to-run sets; check:outstanding-issues, check:ledger-write-discipline, docs:check-links green on tip; check:drift exit 1 by design (the finding) |

docs/outstanding-issues-inbox/6697868a-cc70-48e3-ad6f-598ff4031110.json

@@ -0,0 +1,10 @@
+{
+  "version": 2,
+  "id": "6697868a-cc70-48e3-ad6f-598ff4031110",
+  "createdOn": "2026-08-18",
+  "action": "cancel",
+  "payload": {
+    "requestId": "a1c319d6-a098-460b-8b42-6794ca6690fd",
+    "reason": "Superseded before push: the queued detail recorded check:drift against staging as blocked on a missing service-role key. The key was supplied, the gate ran, and it is red with 19 findings, so that status is wrong. Replaced by a corrected update request in the same PR."
+  }
+}

docs/outstanding-issues-inbox/a1c319d6-a098-460b-8b42-6794ca6690fd.json

@@ -0,0 +1,11 @@
+{
+  "version": 2,
+  "id": "a1c319d6-a098-460b-8b42-6794ca6690fd",
+  "createdOn": "2026-08-18",
+  "action": "update",
+  "payload": {
+    "id": "#056",
+    "detail": "REPLAY COMPLETE 2026-08-18 in an owner-authorized staging window (Phase 2 of docs/database-remediation-plan.md); DRIFT PROOF STILL OUTSTANDING. Target was Clinical KB Staging ref ikoiolksxqxfxgiyqpnu via the Supabase MCP connector, verified on every call; production sjrfecxgysukkwxsowpy was never a mutation target and the only production interaction was list_projects. Gap re-measured at the start of the window as 28, not the 26 recorded 2026-08-17: 166 staging rows against 194 repository migration files, being ten earlier history holes plus eighteen versions after 20260719055623. The full 28-version chain was replayed in version order and staging now holds 194 rows, latest version 20260814151000, zero statements IS NULL rows, with a two-way version diff against supabase/migrations that is empty in both directions. Every replayed row was read back and its md5 compared against md5sum of the repository file: all 28 match byte-for-byte, so the repository versions and text were preserved exactly. supabase db push was unavailable because SUPABASE_ACCESS_TOKEN is still absent (#183) and the staging database password is operator-only, and MCP apply_migration was rejected because it stamps connector-generated timestamps which docs/staging-setup.md forbids; each migration was instead executed verbatim through execute_sql with an explicit schema_migrations row carrying the repository version and name. No production clinical document was copied and no ingestion worker was started: documents and document_chunks remain 0. WHAT REMAINS: check:drift against staging has not been run. Two blockers were found. First, scripts/check-drift.ts passed only three of the five identity keys to checkSupabaseProjectConfig, dropping SUPABASE_STAGING_PROJECT_REF and SUPABASE_STAGING_PROJECT_NAME, so any staging URL was rejected as a production mismatch; that narrowing is fixed in this PR. Second, check:drift authenticates as service_role over PostgREST and no staging service-role key exists in this environment, the MCP connector cannot supply one, and the only local .env.local holds production values, so the gate was deliberately not run rather than risk targeting production. An operator must set the staging URL, ref, name and service-role key locally and run npm run check:drift; expect a first run that is not clean because the manifest is generated from supabase/schema.sql rather than from the migration chain and drift-allowlist.json is empty. SIX FINDINGS from the clean replay are recorded in docs/audit/live-drift-forensics-2026-08.md section Phase 2, none patched. Most consequential: (a) the 20260804110240 guard passed with no prebuild because all four indexes it validates, plus both trigram indexes, already existed on staging valid and ready, so staging replayed the same chain production ran and kept the indexes production lost, which is evidence the production loss was not caused by the committed chain; (b) four migrations exist as duplicate earlier/later version pairs, so a plain supabase db push --include-all re-applies older create-or-replace bodies over newer ones and the later four must be re-executed afterwards; (c) 20260724120000 silently drops the work_mem setting 20260724000000 applies, leaving seven functions with work_mem rather than eight on a clean replay, which bears on the Phase 1.2 hypothesis recorded against #316. #057 soak and rollback stays queued behind the outstanding drift proof.",
+    "baseRowFingerprint": "a3a4b3f5ce5191058beadeb6544917d3fae30259c76d1e17aaea0280e672e3e8"
+  }
+}

docs/outstanding-issues-inbox/d958d671-362d-41ed-b72f-102357dbf156.json

@@ -0,0 +1,11 @@
+{
+  "version": 2,
+  "id": "d958d671-362d-41ed-b72f-102357dbf156",
+  "createdOn": "2026-08-18",
+  "action": "update",
+  "payload": {
+    "id": "#056",
+    "detail": "PHASE 2 RUN 2026-08-18 in an owner-authorized staging window. REPLAY COMPLETE AND PROVEN; check:drift now RUN and RED with 19 findings, which is the substantive result of this phase. Target Clinical KB Staging ref ikoiolksxqxfxgiyqpnu via the Supabase MCP connector, ref re-verified on every call; production sjrfecxgysukkwxsowpy was never a mutation target and the only production interaction was list_projects. GAP re-measured at the start of the window as 28, not the 26 recorded 2026-08-17: 166 staging rows against 194 repository files, being ten earlier history holes plus eighteen versions after 20260719055623. The chain was replayed in version order; staging now holds 194 rows, latest 20260814151000, zero statements IS NULL, and a two-way version diff against supabase/migrations is empty both ways. Every replayed row was read back and md5-compared against the repository file: all 28 match byte-for-byte. supabase db push was unavailable (SUPABASE_ACCESS_TOKEN absent per #183, DB password operator-only) and MCP apply_migration was rejected because it stamps connector-generated versions, which docs/staging-setup.md forbids; each migration ran verbatim through execute_sql with an explicit schema_migrations row carrying the repository version and name. No production clinical document was copied and no worker started: documents and document_chunks remain 0. DRIFT RESULT: 19 unexpected rows, exit 1. Because staging now carries the complete byte-verified chain, this is not staging staleness - it is the committed migration chain and supabase/schema.sql disagreeing, and check:drift builds its expected side from schema.sql. Decomposition: (a) seven match_* def_hash mismatches caused by SET work_mem, which pg_get_functiondef renders and def_hash does not strip; grep -c work_mem supabase/schema.sql returns 0 and only migration 20260724000000 sets it, so schema.sql is the stale side and the fix is repo-side, not a production deploy. The eighth work_mem target, match_document_table_facts_text, is absent from the drift list precisely because 20260724120000 re-created it without restating work_mem, which confirms the mechanism. (b) eight objects schema.sql declares that no migration creates or drops - five document_embedding_fields indexes, documents_status_idx, and the documents_updated_at and ingestion_jobs_updated_at set_updated_at triggers; the two triggers mean updated_at maintenance would silently not exist in any environment built from migrations alone. (c) three table column-set mismatches on document_chunks, rag_visual_eval_cases and rag_visual_eval_runs, not yet expanded per column. (d) one index def mismatch on document_chunks_content_trgm_idx, which is one of the two trigram indexes rebuilt in the 2026-08-14 production incident window and whose canonical definition should be confirmed against what was actually built. PROGRAMME CONSEQUENCE: until schema.sql and the chain are reconciled, a production drift finding cannot be assumed to mean production drifted; for the seven work_mem functions the opposite holds. This argues for reconciling schema.sql to the chain before spending a production window on Phase 3. BEARING ON #316: that row carries the work_mem explanation as an untested hypothesis about production's ten mismatched RPCs; it is now measured on staging for seven of them with no production call. It does not close Phase 1.2 - production reports ten, staging seven, and the residual (match_document_table_facts_text plus the _v2 outliers match_document_chunks_text_v2 and match_document_index_units_hybrid_v2) needs its own diffs. #316 was deliberately not updated from this session. ALSO FIXED HERE: scripts/check-drift.ts forwarded only three of five identity keys to checkSupabaseProjectConfig, so any staging URL resolved to production and was rejected as a mismatch; proven before and after against identical env (before: mismatch/production/sjrfecxgysukkwxsowpy, after: ready/staging/ikoiolksxqxfxgiyqpnu). Six clean-replay findings are recorded in docs/audit/live-drift-forensics-2026-08.md section Phase 2, none patched, no migration edited. #057 soak and rollback is now unblocked on parity grounds, though the drift reconciliation above should land first.",
+    "baseRowFingerprint": "a3a4b3f5ce5191058beadeb6544917d3fae30259c76d1e17aaea0280e672e3e8"
+  }
+}

docs/staging-setup.md

@@ -12,13 +12,20 @@ Staging is two independent tiers: a **staging Supabase project** (data) and a
 > the staging corpus is empty, and `search_schema_health()` passes. Do not create replacements.
 > **Revalidated 2026-07-30:** the staging project and app are still healthy, correctly identify as
 > staging, run with `RAG_PROVIDER_MODE=offline`, and have no OpenAI key. Linked migration history
-> had **24** local-only versions: ten holes before/at `20260719053533` (four are historical
-> placeholders) and fourteen versions after `20260719055623`. **Superseded — remeasured 2026-08-17
-> the chain is 26** (the same ten holes plus **sixteen** after `20260719055623`), and it grows every
-> time `main` advances, so treat any fixed number here as stale and take the count from the dry-run
-> below rather than from this paragraph. See ledger `#056`. `supabase db push --linked
---include-all --dry-run` prints the exact current chain. Do not run a normal or partial push:
-> history is divergent, and the full chain currently ends in the separately governed BMJ
+> had **24** local-only versions on 2026-07-30, remeasured to **26** on 2026-08-17 as `main`
+> advanced. **Resolved 2026-08-18 (Phase 2, ledger `#056`):** the gap measured **28** at the start
+> of the approved staging window (ten history holes plus eighteen versions after `20260719055623`)
+> and the full chain was replayed to parity — staging now holds **194** rows in
+> `supabase_migrations.schema_migrations`, latest `20260814151000`, zero `statements IS NULL`, and a
+> two-way diff against `supabase/migrations/` is empty. Each replayed row was verified byte-identical
+> to its repository file by md5. Evidence and the six replay findings:
+> [`docs/audit/live-drift-forensics-2026-08.md`](audit/live-drift-forensics-2026-08.md) § Phase 2.
+> The chain still grows every time `main` advances, so re-measure rather than trusting any fixed
+> number here. `supabase db push --linked --include-all --dry-run` prints the exact current chain.
+> Do not run a normal or partial push. Note that a plain `--include-all` push is **not** sufficient
+> on its own: four migrations exist as duplicate earlier/later version pairs, and pushing the earlier
+> copies re-applies older `create or replace function` bodies over newer ones — the later four must
+> be re-executed afterwards (Phase 2 finding 2). The chain includes the separately governed BMJ
 > attestation migration `20260727010000`. Reconcile the entire reviewed chain only in an approved
 > scope, then repeat the identity, indexing, health, and empty-data-boundary proof.
 

scripts/check-drift.ts

@@ -339,6 +339,12 @@ async function main() {
       NEXT_PUBLIC_SUPABASE_URL: process.env.NEXT_PUBLIC_SUPABASE_URL,
       SUPABASE_PROJECT_REF: process.env.SUPABASE_PROJECT_REF,
       SUPABASE_PROJECT_NAME: process.env.SUPABASE_PROJECT_NAME,
+      // Staging declarations must be forwarded, or resolveStagingProject() sees
+      // none and every ref is compared against production. check:supabase-project
+      // already passes all five keys; this check must match it so an approved
+      // staging window can run the same drift comparison.
+      SUPABASE_STAGING_PROJECT_REF: process.env.SUPABASE_STAGING_PROJECT_REF,
+      SUPABASE_STAGING_PROJECT_NAME: process.env.SUPABASE_STAGING_PROJECT_NAME,
     },
     { requireMetadata: false },
   );