From 5067c4aad5f19be998381f878a5199f9fcad7230 Mon Sep 17 00:00:00 2001 From: BigSimmo <87357024+BigSimmo@users.noreply.github.com> Date: Tue, 18 Aug 2026 12:33:00 +0800 Subject: [PATCH 1/3] feat(db): replay the full migration chain onto staging to parity (Phase 2, #056) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 2 of docs/database-remediation-plan.md, run in an owner-authorized staging window. Target was Clinical KB Staging (ikoiolksxqxfxgiyqpnu) via the Supabase MCP connector, with the ref re-verified on every call; production (sjrfecxgysukkwxsowpy) was never a mutation target. The gap re-measured as 28 versions at the start of the window, not the 26 recorded on 2026-08-17 — ten earlier history holes plus eighteen versions after 20260719055623. The whole chain was replayed in version order. Staging now holds 194 rows in supabase_migrations.schema_migrations, latest 20260814151000, zero statements IS NULL, and a two-way version diff against supabase/migrations is empty. Every replayed row was read back and its md5 compared against md5sum of the repository file: all 28 match byte-for-byte. supabase db push was unavailable (SUPABASE_ACCESS_TOKEN absent per #183, staging DB password operator-only) and MCP apply_migration was rejected because it stamps connector-generated versions, which docs/staging-setup.md forbids. Each migration was executed verbatim via execute_sql with an explicit schema_migrations row carrying the repository's own version and name. check:drift against staging is not yet proven. scripts/check-drift.ts forwarded only three of the five identity keys to checkSupabaseProjectConfig, so any staging URL was rejected as a production mismatch; that narrowing is fixed here, mirroring scripts/check-supabase-project.ts. The remaining blocker is the staging service-role key, which is not in this environment and which the MCP connector cannot supply. The gate was deliberately not run rather than risk pointing it at production via the only local .env.local. Six findings from the clean replay are recorded, none patched — notably that the 20260804110240 guard passed with no prebuild because staging never lost the indexes production lost, that four migrations exist as duplicate earlier/later version pairs which a plain --include-all push would apply in the wrong effective order, and that 20260724120000 silently drops the work_mem setting 20260724000000 applies. No schema file, migration, or drift manifest was changed. No production clinical document was copied to staging and no ingestion worker was started. --- docs/audit/live-drift-forensics-2026-08.md | 203 +++++++++++++++++- .../a1c319d6-a098-460b-8b42-6794ca6690fd.json | 11 + docs/staging-setup.md | 21 +- scripts/check-drift.ts | 6 + 4 files changed, 230 insertions(+), 11 deletions(-) create mode 100644 docs/outstanding-issues-inbox/a1c319d6-a098-460b-8b42-6794ca6690fd.json diff --git a/docs/audit/live-drift-forensics-2026-08.md b/docs/audit/live-drift-forensics-2026-08.md index 6005d58299..74be25c939 100644 --- a/docs/audit/live-drift-forensics-2026-08.md +++ b/docs/audit/live-drift-forensics-2026-08.md @@ -212,10 +212,205 @@ inventory widens what that pairing has to explain; it does not by itself attribu ## Phase 2 — Staging parity rehearsal -_Not yet run. Requires an approved staging window; production stays read-only._ - -_Pending._ Migration-replay tail, any migration that misbehaved on clean replay (a finding in its -own right), and the green `check:drift` output against staging. +_2026-08-18, owner-authorized staging window ("I authorize mutation of the STAGING Supabase tier +(Clinical KB Staging) only"). Target `Clinical KB Staging`, ref `ikoiolksxqxfxgiyqpnu`, via the +Supabase MCP connector. Production `sjrfecxgysukkwxsowpy` was never a target: the only production +interaction in this window was `list_projects`, and the target ref was restated on every call. +Replay is **complete**; `check:drift` against staging is **blocked** on a credential — see 2.3._ + +### 2.0 Connector and pre-flight + +`list_projects` returned both projects; staging is `ACTIVE_HEALTHY`, Postgres 17.6, +`ap-southeast-2`. The connector is write-capable and connects as `current_user = postgres`, which +is the role hosted migrations target, so `check:migration-role` discipline is preserved. + +Pre-flight checks the replay depended on, all green before any write: + +| Check | Result | +| --------------------------------------------------------- | ---------------------------------------------------------------- | +| `vector`, `pg_trgm`, `uuid-ossp` namespace | all in `extensions` (not `public`) | +| `pg_has_role(current_user,'postgres','MEMBER')` | `true` — the seven default-privilege asserts can pass | +| `default_privileges_status('postgres','public')->>'safe'` | `true` | +| `schema_drift_snapshot()` / `search_schema_health()` | both present (178 KB snapshot, 200 index rows, 87 function rows) | +| `documents` / `document_chunks` row counts | `0` / `0` — staging idle and empty, before and after | + +`pg_net` and `pg_cron` are **not** installed on staging. That did not block the replay (see 2.4) +but it does mean the six retention/purge cron jobs are silently unscheduled there — a parity gap to +record, not a replay failure. + +### 2.1 Before-gap: 28 versions, not 26 + +Re-measured at the start of the window, as `#056` and `docs/staging-setup.md` both instruct. Staging +held **166** rows in `supabase_migrations.schema_migrations`, latest `20260719055623`, with **zero** +`statements IS NULL` rows. The repository holds **194** migration files. The gap was therefore +**28**: **ten earlier history holes** plus **eighteen** versions after `20260719055623` — not the 26 +(ten + sixteen) recorded on 2026-08-17. The two-version increase is `main` advancing while staging +stood still, exactly the widening the ledger predicted. + +The exact missing chain, in version order: + +``` +20260713110000_historical_version_placeholder +20260713120000_historical_version_placeholder +20260713121000_historical_version_placeholder +20260713122000_historical_version_placeholder +20260717133000_historical_version_placeholder +20260717161000_assert_postgres_default_privileges +20260717173000_reassert_postgres_default_privileges +20260718223000_historical_version_placeholder +20260719053532_repair_postgres_default_privileges +20260719053533_enforce_public_title_word_scope +20260719064735_user_account_data_and_admin_uploads +20260719070000_align_existing_acls +20260720170000_add_documents_owner_updated_at_indexed_idx +20260722110000_explicit_document_title_words_backend_policy +20260722190000_bind_publication_approval_to_reviewed_state +20260723150000_document_change_ingestion_webhook +20260724000000_optimize_rpc_work_mem +20260724060000_atomic_reindex_agent_guard +20260724120000_table_facts_plpgsql_execute +20260724130000_explicit_base_match_rpc_execute_grants +20260724130100_fix_invoke_ingestion_worker_url_to_guc +20260724130200_create_uploaded_document_with_ingestion_job +20260725000000_audit_security_remediation +20260727010000_bmj_third_party_source_attestation +20260731150000_db_query_perf_rate_limit_and_image_indexes +20260804110240_restore_rag_search_health_indexes +20260814150000_add_therapy_favourites +20260814151000_validate_therapy_favourites_content_type +``` + +### 2.2 Replay: mechanism, faithfulness proof, and result + +**Mechanism.** `SUPABASE_ACCESS_TOKEN` is still absent (`#183`) and the staging database password is +operator-only, so `supabase db push --linked --include-all` was unavailable. MCP `apply_migration` +was rejected as the substitute because it stamps a connector-generated version, which +`docs/staging-setup.md` explicitly forbids ("do not replay the missing chain through a helper that +records new timestamps"). Each migration was therefore applied through `execute_sql` as a single +implicit transaction that runs the file body verbatim and then writes its own history row carrying +the repository's exact version and name — the same row `supabase db push` would have written. + +**Faithfulness proof.** Every applied row was read back and its `md5(statements[1])` compared +against `md5sum` of the repository file. **All 28 match byte-for-byte**, so the replayed text is +provably the committed text under the repository's own versions: + +``` +=== md5 mismatches (empty = all 28 byte-identical) === +ALL 28 MATCH +``` + +**Result.** Full parity, verified in both directions: + +``` +repo files: 194 staging rows: 194 +--- in repo, missing from staging --- +--- in staging, not in repo --- +--- (both empty = full parity) --- +``` + +`total_rows 194 · latest_version 20260814151000 · no_statements 0` — staging still carries **zero** +`statements IS NULL` rows, so the replay introduced none of the history-repair signal Phase 6.1 is +being built to police. `documents` and `document_chunks` remain `0`: no production clinical document +was copied and no ingestion worker was started. + +**Deviation to record.** The Supabase CLI splits a migration into one array element per statement +(pre-existing staging rows carry 10–19 elements each); these 28 rows store the whole file as a +single element. Version, name, and text are exact — only the array arity differs. Anything that +inspects `cardinality(statements)` will see it, and it is worth normalising if a future tool depends +on per-statement granularity. + +### 2.3 `check:drift` against staging — blocked, with half the blocker fixed here + +Two independent obstacles. One is fixed in this PR; one is an operator credential. + +**Obstacle 1 (fixed here).** `check:drift` could not target staging at all. +`scripts/check-drift.ts` called `checkSupabaseProjectConfig` with only three of the five identity +keys, dropping `SUPABASE_STAGING_PROJECT_REF` and `SUPABASE_STAGING_PROJECT_NAME`. With those +absent, `resolveStagingProject` (`src/lib/supabase/project.ts:95-97`) returns `null`, `expected` +falls back to production, and any staging URL fails as a mismatch — even though `requireServerEnv()` +on the line above is already staging-aware and `scripts/check-supabase-project.ts:6-15` passes all +five. This PR forwards the two keys. It is a latent blocker for every future staging drift check, +not a Phase 2 artifact. + +**Obstacle 2 (open — operator action).** `check:drift` reaches the database as `service_role` over +PostgREST, and no staging service-role key exists in this environment. The Supabase MCP connector +cannot supply one (publishable keys only), and the repository's only `.env.local` holds +**production** values — running the gate with it would have targeted production, which this window +forbids, so it was not run. Capturing the snapshot through the connector instead was considered and +rejected: it would have meant adding a snapshot-input bypass flag to a gate script. + +**To finish.** An operator sets, locally and uncommitted: +`NEXT_PUBLIC_SUPABASE_URL=https://ikoiolksxqxfxgiyqpnu.supabase.co`, `SUPABASE_PROJECT_REF` and +`SUPABASE_STAGING_PROJECT_REF` = `ikoiolksxqxfxgiyqpnu`, `SUPABASE_PROJECT_NAME` and +`SUPABASE_STAGING_PROJECT_NAME` = `Clinical KB Staging`, plus the staging `sb_secret_…` service-role +key; then `npm run check:drift`. Expect the first run **not** to be clean: the manifest is generated +from a from-scratch replay of `supabase/schema.sql`, not of the migration chain, so this comparison +asks the stronger question "do the 194 migrations and `schema.sql` agree?", and +`supabase/drift-allowlist.json` is empty, so every divergence fails rather than warns. + +### 2.4 Findings from the clean replay + +Each is a finding in its own right, per the phase's definition of done. **No migration raised an +error, and no migration file was edited.** + +1. **The `20260804110240` guard passed with no prebuild — and that is evidence about production.** + All four indexes it validates (`document_labels_label_trgm_idx`, + `document_summaries_summary_trgm_idx`, `document_index_units_owner_chunk_type_idx`, + `rag_retrieval_logs_miss_idx`) already existed on staging, `indisvalid` and `indisready`, with + matching definitions — created by `20260705180000` and dropped by no later migration. So did + `documents_title_trgm_idx` and `document_chunks_content_trgm_idx`. Staging replayed the same + chain production ran and **kept** the indexes production lost. That is direct evidence the + production loss was not caused by the committed chain, narrowing `#248` / `#316` attribution + toward an out-of-band drop. Recorded as Phase 2 evidence only — `#316` is owned by the + concurrent Phase 1.2 session and was deliberately not updated from here. + +2. **A version-order inversion exists in the committed chain, and a plain replay handles it wrong.** + The repository carries two near-identical copies of four migrations: + `20260717161000`/`20260717173000`/`20260719053532`/`20260719053533`, re-issued as + `20260719055541`/`055555`/`055609`/`055623`. Staging had applied only the later set, so applying + the earlier set in version order re-ran older `create or replace function` bodies **on top of** + newer ones. The pairs differ only in whitespace except `20260719053533`, whose `raise … hint` + names `20260719053532` where the newer names `20260719055609`. After the four earlier versions + landed, the four later bodies were re-executed from staging's own recorded statements — no + history rows added — so the end state equals a from-scratch ordered replay, and + `default_privileges_status('postgres','public')->>'safe'` is `true` afterwards. A plain + `supabase db push --include-all` would **not** do this and would leave the older bodies live. + +3. **`20260724120000` silently drops the `work_mem` setting `20260724000000` applies.** + `20260724000000_optimize_rpc_work_mem` sets `work_mem = '64MB'` on eight `match_*` functions. + Later in chain order, `20260724120000_table_facts_plpgsql_execute` re-creates + `match_document_table_facts_text` declaring only `set search_path` and `set plan_cache_mode`, and + `CREATE OR REPLACE FUNCTION` resets any config option the new definition does not restate. + Measured on staging after the full replay: **7** functions carry `work_mem`, not 8, and + `match_document_table_facts_text` is the one that does not. This matters to the Phase 1.2 + dossier, whose recorded hypothesis assumes that migration leaves eight functions with `work_mem`; + on a clean chain replay it leaves seven. Not acted on here — protected RAG surface, and `#316` + belongs to the other session. + +4. **`20260724130100` ships a literal `[REDACTED]` placeholder in executable SQL — already + remediated in-chain.** The committed blob contains `'[REDACTED]'` twice where a base URL belongs: + as the `alter database … set app.ingestion_worker_base_url` value, and as the + `invoke_ingestion_worker()` fallback. Confirmed against `git show HEAD:` and not an output mask — + sibling migrations render the real project ref through the same tooling. The next migration, + `20260725000000_audit_security_remediation`, exists precisely to correct it and says so, while + instructing that the applied blob must not be edited. A clean replay therefore self-heals and the + correct action is none. + +5. **A faithful replay points staging's worker GUC at the production project.** `20260725000000` + runs `alter database … set app.ingestion_worker_base_url` to the production URL, and its + `insufficient_privilege` guard does not fire here because the connector is `postgres`. The + setting is database-level, so it applies to new connections. It is inert on staging three times + over — `pg_net` is not installed, the vault secret `cron_ingestion_jwt` is absent (the function + raises `Missing Vault secret` first), and no `pg_cron` schedule exists — but it is a live + cross-environment pointer the moment any of those changes. **Do not seed `cron_ingestion_jwt`, + `indexing_v3_agent_secret`, or `ingestion_webhook_secret` into staging's vault.** By contrast + `20260723150000`'s document-change webhook is safe by construction: it has no production fallback + and returns early when `app.ingestion_webhook_base_url` is unset, which it is. + +6. **Platform-dependent objects applied cleanly despite absent extensions.** `20260723150000` and + `20260724130100` reference `net.http_post` and `vault.decrypted_secrets` from inside PL/pgSQL + bodies, which Postgres does not resolve at `CREATE FUNCTION` time, so both applied without + `pg_net` installed. They fail only when called. ## Phase 3 — RPC reconciliation diff --git a/docs/outstanding-issues-inbox/a1c319d6-a098-460b-8b42-6794ca6690fd.json b/docs/outstanding-issues-inbox/a1c319d6-a098-460b-8b42-6794ca6690fd.json new file mode 100644 index 0000000000..d88680e69b --- /dev/null +++ b/docs/outstanding-issues-inbox/a1c319d6-a098-460b-8b42-6794ca6690fd.json @@ -0,0 +1,11 @@ +{ + "version": 2, + "id": "a1c319d6-a098-460b-8b42-6794ca6690fd", + "createdOn": "2026-08-18", + "action": "update", + "payload": { + "id": "#056", + "detail": "REPLAY COMPLETE 2026-08-18 in an owner-authorized staging window (Phase 2 of docs/database-remediation-plan.md); DRIFT PROOF STILL OUTSTANDING. Target was Clinical KB Staging ref ikoiolksxqxfxgiyqpnu via the Supabase MCP connector, verified on every call; production sjrfecxgysukkwxsowpy was never a mutation target and the only production interaction was list_projects. Gap re-measured at the start of the window as 28, not the 26 recorded 2026-08-17: 166 staging rows against 194 repository migration files, being ten earlier history holes plus eighteen versions after 20260719055623. The full 28-version chain was replayed in version order and staging now holds 194 rows, latest version 20260814151000, zero statements IS NULL rows, with a two-way version diff against supabase/migrations that is empty in both directions. Every replayed row was read back and its md5 compared against md5sum of the repository file: all 28 match byte-for-byte, so the repository versions and text were preserved exactly. supabase db push was unavailable because SUPABASE_ACCESS_TOKEN is still absent (#183) and the staging database password is operator-only, and MCP apply_migration was rejected because it stamps connector-generated timestamps which docs/staging-setup.md forbids; each migration was instead executed verbatim through execute_sql with an explicit schema_migrations row carrying the repository version and name. No production clinical document was copied and no ingestion worker was started: documents and document_chunks remain 0. WHAT REMAINS: check:drift against staging has not been run. Two blockers were found. First, scripts/check-drift.ts passed only three of the five identity keys to checkSupabaseProjectConfig, dropping SUPABASE_STAGING_PROJECT_REF and SUPABASE_STAGING_PROJECT_NAME, so any staging URL was rejected as a production mismatch; that narrowing is fixed in this PR. Second, check:drift authenticates as service_role over PostgREST and no staging service-role key exists in this environment, the MCP connector cannot supply one, and the only local .env.local holds production values, so the gate was deliberately not run rather than risk targeting production. An operator must set the staging URL, ref, name and service-role key locally and run npm run check:drift; expect a first run that is not clean because the manifest is generated from supabase/schema.sql rather than from the migration chain and drift-allowlist.json is empty. SIX FINDINGS from the clean replay are recorded in docs/audit/live-drift-forensics-2026-08.md section Phase 2, none patched. Most consequential: (a) the 20260804110240 guard passed with no prebuild because all four indexes it validates, plus both trigram indexes, already existed on staging valid and ready, so staging replayed the same chain production ran and kept the indexes production lost, which is evidence the production loss was not caused by the committed chain; (b) four migrations exist as duplicate earlier/later version pairs, so a plain supabase db push --include-all re-applies older create-or-replace bodies over newer ones and the later four must be re-executed afterwards; (c) 20260724120000 silently drops the work_mem setting 20260724000000 applies, leaving seven functions with work_mem rather than eight on a clean replay, which bears on the Phase 1.2 hypothesis recorded against #316. #057 soak and rollback stays queued behind the outstanding drift proof.", + "baseRowFingerprint": "a3a4b3f5ce5191058beadeb6544917d3fae30259c76d1e17aaea0280e672e3e8" + } +} diff --git a/docs/staging-setup.md b/docs/staging-setup.md index 5f0d872484..d074b98281 100644 --- a/docs/staging-setup.md +++ b/docs/staging-setup.md @@ -12,13 +12,20 @@ Staging is two independent tiers: a **staging Supabase project** (data) and a > the staging corpus is empty, and `search_schema_health()` passes. Do not create replacements. > **Revalidated 2026-07-30:** the staging project and app are still healthy, correctly identify as > staging, run with `RAG_PROVIDER_MODE=offline`, and have no OpenAI key. Linked migration history -> had **24** local-only versions: ten holes before/at `20260719053533` (four are historical -> placeholders) and fourteen versions after `20260719055623`. **Superseded — remeasured 2026-08-17 -> the chain is 26** (the same ten holes plus **sixteen** after `20260719055623`), and it grows every -> time `main` advances, so treat any fixed number here as stale and take the count from the dry-run -> below rather than from this paragraph. See ledger `#056`. `supabase db push --linked ---include-all --dry-run` prints the exact current chain. Do not run a normal or partial push: -> history is divergent, and the full chain currently ends in the separately governed BMJ +> had **24** local-only versions on 2026-07-30, remeasured to **26** on 2026-08-17 as `main` +> advanced. **Resolved 2026-08-18 (Phase 2, ledger `#056`):** the gap measured **28** at the start +> of the approved staging window (ten history holes plus eighteen versions after `20260719055623`) +> and the full chain was replayed to parity — staging now holds **194** rows in +> `supabase_migrations.schema_migrations`, latest `20260814151000`, zero `statements IS NULL`, and a +> two-way diff against `supabase/migrations/` is empty. Each replayed row was verified byte-identical +> to its repository file by md5. Evidence and the six replay findings: +> [`docs/audit/live-drift-forensics-2026-08.md`](audit/live-drift-forensics-2026-08.md) § Phase 2. +> The chain still grows every time `main` advances, so re-measure rather than trusting any fixed +> number here. `supabase db push --linked --include-all --dry-run` prints the exact current chain. +> Do not run a normal or partial push. Note that a plain `--include-all` push is **not** sufficient +> on its own: four migrations exist as duplicate earlier/later version pairs, and pushing the earlier +> copies re-applies older `create or replace function` bodies over newer ones — the later four must +> be re-executed afterwards (Phase 2 finding 2). The chain includes the separately governed BMJ > attestation migration `20260727010000`. Reconcile the entire reviewed chain only in an approved > scope, then repeat the identity, indexing, health, and empty-data-boundary proof. diff --git a/scripts/check-drift.ts b/scripts/check-drift.ts index a6b7a3004b..64004a5d65 100644 --- a/scripts/check-drift.ts +++ b/scripts/check-drift.ts @@ -216,6 +216,12 @@ async function main() { NEXT_PUBLIC_SUPABASE_URL: process.env.NEXT_PUBLIC_SUPABASE_URL, SUPABASE_PROJECT_REF: process.env.SUPABASE_PROJECT_REF, SUPABASE_PROJECT_NAME: process.env.SUPABASE_PROJECT_NAME, + // Staging declarations must be forwarded, or resolveStagingProject() sees + // none and every ref is compared against production. check:supabase-project + // already passes all five keys; this check must match it so an approved + // staging window can run the same drift comparison. + SUPABASE_STAGING_PROJECT_REF: process.env.SUPABASE_STAGING_PROJECT_REF, + SUPABASE_STAGING_PROJECT_NAME: process.env.SUPABASE_STAGING_PROJECT_NAME, }, { requireMetadata: false }, ); From 523f8a55b36f7ef1a7b2d149d3030cb768f95dc6 Mon Sep 17 00:00:00 2001 From: BigSimmo <87357024+BigSimmo@users.noreply.github.com> Date: Tue, 18 Aug 2026 13:25:11 +0800 Subject: [PATCH 2/3] =?UTF-8?q?docs(db):=20record=20the=20staging=20drift?= =?UTF-8?q?=20result=20=E2=80=94=20the=20chain=20does=20not=20reproduce=20?= =?UTF-8?q?schema.sql?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit check:drift now runs against staging and is red with 19 findings. Because staging carries the complete, byte-verified migration chain, this is not staging staleness: it is the committed chain and supabase/schema.sql disagreeing, and check:drift builds its expected side from schema.sql. Seven match_* def_hash mismatches are SET work_mem, which pg_get_functiondef renders and def_hash does not strip. grep -c work_mem supabase/schema.sql returns 0 and only 20260724000000 sets it, so schema.sql is the stale side and the fix is repo-side, not a production deploy. The eighth work_mem target, match_document_table_facts_text, is absent from the drift list precisely because 20260724120000 re-created it without restating work_mem — the exception confirms the mechanism. Eight further objects are declared in schema.sql that no migration creates or drops: five document_embedding_fields indexes, documents_status_idx, and the documents_updated_at and ingestion_jobs_updated_at set_updated_at triggers. Three table column-set mismatches and one index def mismatch remain unclassified. This bears on #316's recorded work_mem hypothesis, now measured on staging for seven of production's ten mismatched RPCs with no production call. It does not close Phase 1.2 and #316 was deliberately not updated from this session. The superseded #056 inbox request is cancelled and replaced rather than edited, per the immutable-request rule. Measurement base: ed43a64f2. origin/main has since advanced to 6b72b3a04 (195 migrations, including a schema_drift_snapshot history probe); re-measuring against that manifest is recorded as follow-up. --- docs/audit/live-drift-forensics-2026-08.md | 128 ++++++++++++++---- .../6697868a-cc70-48e3-ad6f-598ff4031110.json | 10 ++ .../d958d671-362d-41ed-b72f-102357dbf156.json | 11 ++ 3 files changed, 120 insertions(+), 29 deletions(-) create mode 100644 docs/outstanding-issues-inbox/6697868a-cc70-48e3-ad6f-598ff4031110.json create mode 100644 docs/outstanding-issues-inbox/d958d671-362d-41ed-b72f-102357dbf156.json diff --git a/docs/audit/live-drift-forensics-2026-08.md b/docs/audit/live-drift-forensics-2026-08.md index 74be25c939..757f55f536 100644 --- a/docs/audit/live-drift-forensics-2026-08.md +++ b/docs/audit/live-drift-forensics-2026-08.md @@ -216,7 +216,7 @@ _2026-08-18, owner-authorized staging window ("I authorize mutation of the STAGI (Clinical KB Staging) only"). Target `Clinical KB Staging`, ref `ikoiolksxqxfxgiyqpnu`, via the Supabase MCP connector. Production `sjrfecxgysukkwxsowpy` was never a target: the only production interaction in this window was `list_projects`, and the target ref was restated on every call. -Replay is **complete**; `check:drift` against staging is **blocked** on a credential — see 2.3._ +Replay is **complete**. `check:drift` against staging has now **run** and is **red with 19 findings** — see 2.3._ ### 2.0 Connector and pre-flight @@ -319,34 +319,104 @@ single element. Version, name, and text are exact — only the array arity diffe inspects `cardinality(statements)` will see it, and it is worth normalising if a future tool depends on per-statement granularity. -### 2.3 `check:drift` against staging — blocked, with half the blocker fixed here - -Two independent obstacles. One is fixed in this PR; one is an operator credential. - -**Obstacle 1 (fixed here).** `check:drift` could not target staging at all. -`scripts/check-drift.ts` called `checkSupabaseProjectConfig` with only three of the five identity -keys, dropping `SUPABASE_STAGING_PROJECT_REF` and `SUPABASE_STAGING_PROJECT_NAME`. With those -absent, `resolveStagingProject` (`src/lib/supabase/project.ts:95-97`) returns `null`, `expected` -falls back to production, and any staging URL fails as a mismatch — even though `requireServerEnv()` -on the line above is already staging-aware and `scripts/check-supabase-project.ts:6-15` passes all -five. This PR forwards the two keys. It is a latent blocker for every future staging drift check, -not a Phase 2 artifact. - -**Obstacle 2 (open — operator action).** `check:drift` reaches the database as `service_role` over -PostgREST, and no staging service-role key exists in this environment. The Supabase MCP connector -cannot supply one (publishable keys only), and the repository's only `.env.local` holds -**production** values — running the gate with it would have targeted production, which this window -forbids, so it was not run. Capturing the snapshot through the connector instead was considered and -rejected: it would have meant adding a snapshot-input bypass flag to a gate script. - -**To finish.** An operator sets, locally and uncommitted: -`NEXT_PUBLIC_SUPABASE_URL=https://ikoiolksxqxfxgiyqpnu.supabase.co`, `SUPABASE_PROJECT_REF` and -`SUPABASE_STAGING_PROJECT_REF` = `ikoiolksxqxfxgiyqpnu`, `SUPABASE_PROJECT_NAME` and -`SUPABASE_STAGING_PROJECT_NAME` = `Clinical KB Staging`, plus the staging `sb_secret_…` service-role -key; then `npm run check:drift`. Expect the first run **not** to be clean: the manifest is generated -from a from-scratch replay of `supabase/schema.sql`, not of the migration chain, so this comparison -asks the stronger question "do the 194 migrations and `schema.sql` agree?", and -`supabase/drift-allowlist.json` is empty, so every divergence fails rather than warns. +### 2.3 `check:drift` against staging — RUN, and red: the chain does not reproduce `schema.sql` + +**Blocker cleared.** `check:drift` previously could not target staging at all. +`scripts/check-drift.ts` passed only three of the five identity keys to +`checkSupabaseProjectConfig`, dropping `SUPABASE_STAGING_PROJECT_REF` and +`SUPABASE_STAGING_PROJECT_NAME`; with those absent `resolveStagingProject` +(`src/lib/supabase/project.ts:95-97`) returns `null`, `expected` falls back to production, and any +staging URL is rejected. Fixed here by forwarding both keys, mirroring +`scripts/check-supabase-project.ts:6-15`. Proven against the same env before and after: + +``` +AFTER FIX -> ready | environment: staging | expected: ikoiolksxqxfxgiyqpnu +BEFORE FIX -> mismatch | environment: production | expected: sjrfecxgysukkwxsowpy +``` + +**Result: 19 unexpected drift rows, exit 1.** Staging carries the complete, byte-verified migration +chain, so this is not staging being stale — it is the committed chain and `supabase/schema.sql` +disagreeing: + +``` +Drift manifest: generated 2026-08-16T14:37:41.042Z from schema.sql 365e3368a47b… +Compared 6 extensions, 38 tables, 1 views, 93 functions, 210 indexes, 48 policies, +170 constraints, 26 triggers, 2 storage_buckets against live. + +UNEXPECTED DRIFT (19): +``` + +The 19 decompose into four groups. + +**(a) Seven `match_*` `def_hash` mismatches — `SET work_mem`, non-behavioural, and `schema.sql` is +the stale side.** `20260724000000_optimize_rpc_work_mem` applies `SET work_mem = '64MB'` to eight +`match_*` functions. `pg_get_functiondef` renders function `SET` attributes, and +`schema_drift_snapshot()`'s `def_hash` strips comments and whitespace but **not** `SET` clauses — so +a function carrying `work_mem` cannot hash-match one that does not. + +Measured, offline and decisive: `grep -c work_mem supabase/schema.sql` returns **0**, and the only +migration mentioning it is `20260724000000`. After the full replay staging has **7** functions +carrying `work_mem`, and those 7 are exactly the 7 mismatching here: +`match_document_chunks_hybrid`, `match_document_chunks_text`, +`match_document_embedding_fields_hybrid`, `match_document_index_units_hybrid`, +`match_document_lookup_chunks_text`, `match_document_memory_cards_hybrid`, +`match_document_memory_cards_hybrid_v2`. The eighth work_mem target, +`match_document_table_facts_text`, **does not appear in the drift list** — because +`20260724120000` re-created it without restating `work_mem` (finding 3 above) and it therefore +matches `schema.sql`. The exception proves the mechanism. + +This is a planner memory setting: it affects latency, not row content or ordering. The correct +disposition is to update `supabase/schema.sql` and regenerate `drift-manifest.json` — **a repo-side +fix, not a production deploy.** + +**Bearing on `#316`/Phase 1.2 — recorded, not acted on.** `#316` carries this as an untested +hypothesis about production's 10 mismatched RPCs. It is now measured on staging for 7 of them, from +the committed chain alone, with no production call. It does **not** close Phase 1.2: production +reports 10, staging 7, and the residual — `match_document_table_facts_text` plus the two `_v2` +outliers `match_document_chunks_text_v2` and `match_document_index_units_hybrid_v2` — is not +explained by this mechanism and still needs its own diff. Phase 1.2 owns that; `#316` was not +updated from here. + +**(b) Eight objects `schema.sql` declares that no migration creates.** Verified by grepping every +migration for both `create` and `drop` of each name: there is **no** creating migration and **no** +dropping migration. They exist only in `schema.sql`. + +| Object | Kind | +| --------------------------------------------------------- | ------- | +| `document_embedding_fields_meta_rag_indexing_version_idx` | index | +| `document_embedding_fields_owner_document_created_idx` | index | +| `document_embedding_fields_owner_id_idx` | index | +| `document_embedding_fields_search_tsv_chunk_gin_idx` | index | +| `document_embedding_fields_source_chunk_id_idx` | index | +| `documents_status_idx` | index | +| `documents.documents_updated_at` | trigger | +| `ingestion_jobs.ingestion_jobs_updated_at` | trigger | + +(The `ingestion_jobs_updated_at` string does appear in `20260712170500`, but as +`ingestion_jobs_updated_at_idx`, a different object.) Five of these are on +`document_embedding_fields`, a retrieval-path table, and two are `set_updated_at` triggers whose +absence would silently stop `updated_at` maintenance on `documents` and `ingestion_jobs` in any +environment built from migrations alone. + +**(c) Three table column-set mismatches:** `document_chunks`, `rag_visual_eval_cases`, +`rag_visual_eval_runs`. The truncated diff needs per-column expansion before classification; not +attempted here. + +**(d) One index definition mismatch:** `document_chunks_content_trgm_idx` — `def_hash` +manifest `8499c3d3…` vs live `c3db2960…`. Note this is one of the two trigram indexes restored on +production in the 2026-08-14 incident window, so its canonical definition is worth confirming +against what was actually built there. + +**What this means for the programme.** `check:drift`'s expected side is generated from +`supabase/schema.sql`, not from the migration chain, and this run is the first end-to-end evidence +that the two disagree in 19 places. Until they are reconciled, a production drift finding cannot be +assumed to mean "production drifted" — for at least the seven work_mem functions the opposite is +true, and `schema.sql` is wrong. That materially changes how Phase 3's classifications should be +read, and it is an argument for reconciling `schema.sql` to the chain **before** spending a +production window. + +None of it was fixed here: this is a docs/evidence PR, and every disposition above is either +repo-side work for another change or Phase 1.2/3 territory. ### 2.4 Findings from the clean replay diff --git a/docs/outstanding-issues-inbox/6697868a-cc70-48e3-ad6f-598ff4031110.json b/docs/outstanding-issues-inbox/6697868a-cc70-48e3-ad6f-598ff4031110.json new file mode 100644 index 0000000000..3f79959fc3 --- /dev/null +++ b/docs/outstanding-issues-inbox/6697868a-cc70-48e3-ad6f-598ff4031110.json @@ -0,0 +1,10 @@ +{ + "version": 2, + "id": "6697868a-cc70-48e3-ad6f-598ff4031110", + "createdOn": "2026-08-18", + "action": "cancel", + "payload": { + "requestId": "a1c319d6-a098-460b-8b42-6794ca6690fd", + "reason": "Superseded before push: the queued detail recorded check:drift against staging as blocked on a missing service-role key. The key was supplied, the gate ran, and it is red with 19 findings, so that status is wrong. Replaced by a corrected update request in the same PR." + } +} diff --git a/docs/outstanding-issues-inbox/d958d671-362d-41ed-b72f-102357dbf156.json b/docs/outstanding-issues-inbox/d958d671-362d-41ed-b72f-102357dbf156.json new file mode 100644 index 0000000000..a78792469a --- /dev/null +++ b/docs/outstanding-issues-inbox/d958d671-362d-41ed-b72f-102357dbf156.json @@ -0,0 +1,11 @@ +{ + "version": 2, + "id": "d958d671-362d-41ed-b72f-102357dbf156", + "createdOn": "2026-08-18", + "action": "update", + "payload": { + "id": "#056", + "detail": "PHASE 2 RUN 2026-08-18 in an owner-authorized staging window. REPLAY COMPLETE AND PROVEN; check:drift now RUN and RED with 19 findings, which is the substantive result of this phase. Target Clinical KB Staging ref ikoiolksxqxfxgiyqpnu via the Supabase MCP connector, ref re-verified on every call; production sjrfecxgysukkwxsowpy was never a mutation target and the only production interaction was list_projects. GAP re-measured at the start of the window as 28, not the 26 recorded 2026-08-17: 166 staging rows against 194 repository files, being ten earlier history holes plus eighteen versions after 20260719055623. The chain was replayed in version order; staging now holds 194 rows, latest 20260814151000, zero statements IS NULL, and a two-way version diff against supabase/migrations is empty both ways. Every replayed row was read back and md5-compared against the repository file: all 28 match byte-for-byte. supabase db push was unavailable (SUPABASE_ACCESS_TOKEN absent per #183, DB password operator-only) and MCP apply_migration was rejected because it stamps connector-generated versions, which docs/staging-setup.md forbids; each migration ran verbatim through execute_sql with an explicit schema_migrations row carrying the repository version and name. No production clinical document was copied and no worker started: documents and document_chunks remain 0. DRIFT RESULT: 19 unexpected rows, exit 1. Because staging now carries the complete byte-verified chain, this is not staging staleness - it is the committed migration chain and supabase/schema.sql disagreeing, and check:drift builds its expected side from schema.sql. Decomposition: (a) seven match_* def_hash mismatches caused by SET work_mem, which pg_get_functiondef renders and def_hash does not strip; grep -c work_mem supabase/schema.sql returns 0 and only migration 20260724000000 sets it, so schema.sql is the stale side and the fix is repo-side, not a production deploy. The eighth work_mem target, match_document_table_facts_text, is absent from the drift list precisely because 20260724120000 re-created it without restating work_mem, which confirms the mechanism. (b) eight objects schema.sql declares that no migration creates or drops - five document_embedding_fields indexes, documents_status_idx, and the documents_updated_at and ingestion_jobs_updated_at set_updated_at triggers; the two triggers mean updated_at maintenance would silently not exist in any environment built from migrations alone. (c) three table column-set mismatches on document_chunks, rag_visual_eval_cases and rag_visual_eval_runs, not yet expanded per column. (d) one index def mismatch on document_chunks_content_trgm_idx, which is one of the two trigram indexes rebuilt in the 2026-08-14 production incident window and whose canonical definition should be confirmed against what was actually built. PROGRAMME CONSEQUENCE: until schema.sql and the chain are reconciled, a production drift finding cannot be assumed to mean production drifted; for the seven work_mem functions the opposite holds. This argues for reconciling schema.sql to the chain before spending a production window on Phase 3. BEARING ON #316: that row carries the work_mem explanation as an untested hypothesis about production's ten mismatched RPCs; it is now measured on staging for seven of them with no production call. It does not close Phase 1.2 - production reports ten, staging seven, and the residual (match_document_table_facts_text plus the _v2 outliers match_document_chunks_text_v2 and match_document_index_units_hybrid_v2) needs its own diffs. #316 was deliberately not updated from this session. ALSO FIXED HERE: scripts/check-drift.ts forwarded only three of five identity keys to checkSupabaseProjectConfig, so any staging URL resolved to production and was rejected as a mismatch; proven before and after against identical env (before: mismatch/production/sjrfecxgysukkwxsowpy, after: ready/staging/ikoiolksxqxfxgiyqpnu). Six clean-replay findings are recorded in docs/audit/live-drift-forensics-2026-08.md section Phase 2, none patched, no migration edited. #057 soak and rollback is now unblocked on parity grounds, though the drift reconciliation above should land first.", + "baseRowFingerprint": "a3a4b3f5ce5191058beadeb6544917d3fae30259c76d1e17aaea0280e672e3e8" + } +} From a6daac6c6cc7b0f925054b8255d2e5a60e080bef Mon Sep 17 00:00:00 2001 From: BigSimmo <87357024+BigSimmo@users.noreply.github.com> Date: Tue, 18 Aug 2026 13:28:56 +0800 Subject: [PATCH 3/3] docs(ledger): record the Phase 2 staging-parity review --- ...48e6ced1f60805bc47334ea1e7147a46a5f13b0b8e4e7ef818e.record.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 docs/branch-review-records/d54d14f00e65448e6ced1f60805bc47334ea1e7147a46a5f13b0b8e4e7ef818e.record.md diff --git a/docs/branch-review-records/d54d14f00e65448e6ced1f60805bc47334ea1e7147a46a5f13b0b8e4e7ef818e.record.md b/docs/branch-review-records/d54d14f00e65448e6ced1f60805bc47334ea1e7147a46a5f13b0b8e4e7ef818e.record.md new file mode 100644 index 0000000000..3f94f465ce --- /dev/null +++ b/docs/branch-review-records/d54d14f00e65448e6ced1f60805bc47334ea1e7147a46a5f13b0b8e4e7ef818e.record.md @@ -0,0 +1 @@ +| 2026-08-18 | claude/db-remediation-phase-2-a0c20b | 523f8a55b36f7ef1a7b2d149d3030cb768f95dc6 | Phase 2 staging parity replay (#056): 28-migration chain replayed onto Clinical KB Staging via authorized MCP connector with md5 byte-verification; scripts/check-drift.ts staging-key fix; check:drift run against staging (red, 19 findings); forensics Phase 2 section; two #056 inbox requests (one cancel, one update) | Self-review passed. Replay proven: 194/194 parity, zero statements IS NULL, all 28 rows md5-identical to repo files, staging corpus untouched (0 documents). Drift check red with 19 findings, correctly interpreted as chain-vs-schema.sql divergence rather than staging staleness; nothing patched. No migration, schema.sql or drift-manifest changed. Production never a mutation target. Known scope limit: measured at base ed43a64f2; origin/main has since advanced to 195 migrations including a schema_drift_snapshot history probe, so a re-measure is owed and is stated in the PR body. | verify:pr-local (green through lint+typecheck); npm run test 2 pre-existing unrelated failures with disjoint run-to-run sets; check:outstanding-issues, check:ledger-write-discipline, docs:check-links green on tip; check:drift exit 1 by design (the finding) |