diff --git a/.cursor/agents/pr-babysit.md b/.cursor/agents/pr-babysit.md new file mode 100644 index 0000000000..c004a5663e --- /dev/null +++ b/.cursor/agents/pr-babysit.md @@ -0,0 +1,20 @@ +--- +name: pr-babysit +description: Get an open Database PR merge-ready by resolving merge conflicts, failing required CI, and actionable unresolved review threads. Use proactively when asked to babysit a PR, land a PR, or clear merge blockers. +--- + +You are a PR babysitter for the Database repository. + +When invoked: + +1. Identify the target PR (number, URL, or current branch) and fetch mergeability, check rollup, and unresolved review threads only. +2. Prefer the smallest safe fix that unblocks merge. Do not rewrite unrelated code. +3. Merge `origin/main` into the feature branch when the PR is behind or conflicts are trivial; abort and ask when intents conflict. +4. Fix CI failures caused by this PR’s scope. Never weaken workflows or delete required checks to force green. +5. Treat Codex/Bugbot findings as actionable only when validated against the current diff; fix clear P0/P1 and scoped P2s, otherwise reply with a concise disposition. +6. After fixing a review thread, reply with `` as the first line when using the repo’s resolve workflow, then resolve the thread when tooling allows. +7. Respect provider confirmation boundaries: no live Supabase/OpenAI/eval spend unless the user explicitly authorized it for this PR. +8. Push ordinary commits to the feature branch, update the PR body when policy metadata is missing (Clinical Governance Preflight, RAG impact, verification), and re-check CI until mergeable + green + comments triaged. +9. Never merge into `main`, force-push, close the PR, or delete branches unless the user explicitly asks. + +Report before/after: merge state, CI, threads fixed vs left open, commits pushed, and any remaining human decision. diff --git a/docs/branch-review-ledger.md b/docs/branch-review-ledger.md index 9f2bee6734..03d3fca652 100644 --- a/docs/branch-review-ledger.md +++ b/docs/branch-review-ledger.md @@ -20,6 +20,8 @@ Use this ledger to prevent repeated branch and PR reviews when the reviewed HEAD | Date | Branch or ref | Reviewed HEAD | Scope | Outcome | Checks | | ---------- | -------------------------------------------------------- | ---------------------------------------- | ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| 2026-07-24 | `cursor/database-interface-audit-0883` / PR #1133 | `8c4c5556ef470673da492aa5f901513c84637d83` | PR babysit + Bugbot + Codex thread triage | COMPLETED for current head. Fixed Codex P2s: stranded queued recovery pages past open-job rows; bulk retry_failed enrichment lease preflight scopes to failed docs only. Bugbot ClinicalDashboard safety-findings finding is not in this PR unique diff vs main. PR policy Clinical Governance Preflight added in body. Merged origin/main. | Local Bugbot; focused Vitest; gh PR/CI. | +| 2026-07-24 | `main` | `6ceaaff50712e10e857bf9a5a7ec88b530bf7b35` | Supabase interface / performance / schema guardian audit | COMPLETED. No P0/P1 live security hole. Confirmed service-role + app-layer ownership model, fail-closed `retrieval_owner_matches`, and project-ref pinning. P2 findings: duplicate unscoped `correct_clinical_query_terms` block in `schema.sql` (safe definition wins at replay); reindex routes miss fresh enrichment-lease gate (`#052`); upload crash can strand `queued` without a job (`#062`); table-facts RPC still `LANGUAGE sql` + `force_custom_plan` (byte-identical plpgsql+EXECUTE remains the latency win). P3: base match RPC execute revokes rely on roles.sql; `invoke_ingestion_worker` hardcodes URL; cold multi-RPC fan-out. Remediation continues on `cursor/database-interface-audit-0883`. | Static schema/RLS/RPC/grant/owner-scope/auth/client inspection; upload/reindex wiring; scale/SLO/deploy docs; outstanding-issues `#052`/`#062`. Provider-gated skipped: `check:supabase-project`, live `check:drift`, `check:indexing`, `profile:retrieval`, `eval:retrieval*`, migration apply. Notion MCP unavailable (`needsAuth`). | | 2026-07-24 | PR #1125 / `codex/answer-relevance-fail-closed` | `8d9fb2408f13e305138749655214baa0020fcfd4` | Follow-up: clear comparison/`documentBreakdown` in untrusted clinical notes | APPROVE for the scoped P2. `trustGatedAnswerForClinicalNotes` now clears `documentBreakdown`, `comparisonMatrix`, and `comparisonEvaluationState` when relevance is not source-backed, so Clinical Notes → ClinicalOutputPanel cannot rebuild comparison-detail tables from raw `best_quote` values. Prior visual/section/quote gates remain. Residual risk is still deliberate low-trust rendering for legacy payloads without `isSourceBacked: true`. | Focused jsdom/policy regressions: `tests/visual-evidence-tabs.dom.test.tsx` 5/5 after hardening the comparison case (caption + matrix values absent). Thread disposition posted and resolved. No live RAG/OpenAI/Supabase mutation. | | 2026-07-24 | `codex/answer-relevance-fail-closed` | `ac0d4305478a0bc8fef03894b78ec5911912c08a` | Missing answer-relevance metadata across render policy and live dashboard grounding | APPROVE after resolving two review P2s. A shared `isAnswerSourceBacked` predicate now requires explicit `isSourceBacked: true`; missing or explicitly negative relevance cannot retain high render trust, a grounded dashboard state, visual/table evidence, or a clinical-notes table bypass. Explicitly source-backed answers preserve supported behavior. Retrieval, ranking, generation, source selection and stored data are unchanged. Highest residual risk is deliberate compatibility tightening for older answer payloads without relevance metadata; they render low-trust and expose review sources rather than richer evidence blocks. | Initial red policy proof failed with `high`; two later red proofs exposed retained visual evidence and the clinical-notes raw-table affordance, then passed after both render-model gates. Focused render/provenance/clinical-safety tests passed 37/37; the focused DOM/policy pair passed 31/31. Offline RAG passed 21 suites/308 tests and 36/36 fixtures; production-readiness was READY against `Clinical KB Database` read-only; `verify:pr-local` passed runtime, formatting, lint, typecheck, all 366 test files (3,254 passed/1 skipped), production build (1,677 pages), client-bundle secret scan and RAG fixture validation; the earlier local `verify:ui` passed 267/267. After the final UI fix, `verify:cheap` again passed all 20 non-test gates, lint, typecheck and 3,254 tests, with only tracked issue #067 timing out under machine load; its isolated retry also exceeded the same 30-second limit and was not repeated. Fresh exact-head hosted checks are required. No live RAG, OpenAI request, Supabase mutation, Railway action, production data operation or deployment ran. | | 2026-07-24 | `codex/safety-plan-no-patient-data-contract` | `b94987c94537f3114a3429848fa908bdecd1d80a` | Safety Plan Generator identifier, local-state, copy, print, privacy-notice and PIA contract | APPROVE. No P0-P2 finding. The patient name/initials field is removed; the builder now asks for identifier-free minimum content, retains working state only in the mounted React component, and makes clipboard/print/PDF export an explicit handling boundary. The PIA and product privacy copy distinguish this local-only tool from provider-backed questions. Highest residual risk is outside Clinical KB: users must handle exported copies under an approved clinical-record process, which the UI now states at the export controls. | Privacy/component DOM 3/3 plus updated privacy-copy 2/2; focused Chromium copy/print/no-fetch-or-XHR 1/1; `verify:cheap` passed all 21 gates, 366 files and 3,245 tests with 1 skip; production-readiness READY using the existing canonical environment without a provider call; production build and client-bundle secret scan passed; offline RAG fixture/manifest 36 cases/21 suites passed. `verify:pr-local` passed runtime, formatting, lint and typecheck, then stopped on the unrelated load-sensitive `reconciliation-preflight` 30-second timeout; that test passed 5/5 isolated and the preceding full suite passed, so the unchanged five-minute gate was not retried. No Supabase, OpenAI, Railway, live RAG, production data or deployment action ran. | diff --git a/docs/outstanding-issues.md b/docs/outstanding-issues.md index e7da0aa5a6..f3917b3981 100644 --- a/docs/outstanding-issues.md +++ b/docs/outstanding-issues.md @@ -48,45 +48,43 @@ removed after current-main verification; it is not missing recommended work. database/RAG/clinical/privacy expertise; Operator = named provider/product/legal authority. - **Estimate:** focused active time, excluding approval, hosted runtime, soak, and review waits. -| Order | ID(s) | Acuity | Capability | When | Estimate | Outcome, gate, verification, and stopping condition | -| ----: | ---------------------- | -------- | ------------------------------------------- | ------------------------------------------------------------------ | ---------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| 1 | `#059` | A1 | Operator security + independent reviewer | Immediate approved security window | 1–3 hours plus verification | Verify every reported exposed credential (GitHub, OpenAI, Supabase service role/database, E2E) is retired; rotate anything still valid and update only intended secret stores. Never record values; stop before provider action without approval. | -| 2 | `#053` | A1 | Operator — legal/privacy | Start now; finish before real patient use/privacy-approved release | 4–8 hours internal; 1–6 weeks elapsed | Execute DPAs; decide ZDR/residency; obtain cache behavior in writing; review subprocessors; obtain APP 8 and APP 5/1 counsel sign-off. Do not change public copy before approval. | -| 4 | `#052` | A2 | High — ingestion concurrency | Ready; next code task | 0.5–1 day | Add red single/bulk tests, then block full/retry reindex during a fresh agent-enrichment lease while keeping stale leases and enrichment mode unchanged. Run focused safety tests and `verify:cheap`; stop if current `main` cannot reproduce it. | -| 5 | `#062` | A2 | Specialist — queue reliability | Ready; after `#052` | 0.5–1.5 days | Add a stranded-row reproducer, then implement the smallest idempotent owner-scoped recovery path for aged queued documents with no open job. Stop if safe age/ownership cannot be proved; hosted changes require approval. | -| 6 | `#066` | A2 | High — frontend release | Ready; after `#062` | 2–4 hours | Finish sidebar local/browser QA, `verify:ui`, production build, and bundle checks, then merge only with green required checks and exact `origin/main` proof. Stop on responsive, accessibility, focus, or performance regression. | -| 7 | `#067` | A3 | High — test reliability | Next flake-hardening window | 1–2 hours | Reproduce the load-sensitive reconciliation-preflight subprocess timeout, instrument its lifecycle, and make the smallest deterministic harness fix. Do not raise the global timeout or bypass the shared heavy-test lock without causal proof. | -| 8 | `#030` | A2 | High — evaluation semantics | Decision-ready; after `#062` | 2–4 hours | Require distinct source identities for distinct comparison slots. Run focused matching tests, typecheck, and `verify:cheap`; stop without changing strict aliases, retrieval, or ranking. | -| 9 | `#019` | A2 | Specialist — RAG answer pipeline | Local reproducer now; behavior change after `#051`/`#023` | 0.5–1 day reproducer | Reproduce admission-source loss in the fallback layer using PR #1096’s source shape. Any behavior change needs protected review and an approved baseline/post canary; stop if independently non-reproducible. | -| 10 | `#054` | A2 | Standard locally; Operator hosted | Local safety identifier now; hosted next approved window | 15–30 min local; 1–2 hours hosted | Presence-check and fill confirmed secret/config gaps with distinct per-environment values. Never record values. Require clean readiness/secret checks; stop on ambiguous environment or project identity. | -| 11 | `#022` | A2 | Operator — clinical governance + Specialist | Decision-ready | 1–2 hours policy; 0.5–1 day first ten | Decide BMJ attestation policy and review the ten highest-impact local documents. Record reviewer/evidence/time; stop after ten and remeasure warning debt. | -| 12 | `#051`, `#023` | A2 | Specialist — RAG diagnostics | After scheduled 2026-07-26 run | 2–4 hours | With GitHub-read approval, compare structured canary/browser/irrelevant-at-10 artifacts without dispatching a rerun. Record deterministic/provider/latency deltas and disposition residuals; stop without spending. | -| 13 | `#018` | A2 | Specialist — clinical RAG/retrieval | After `#051`/`#023`, one mechanism at a time | 1–2 days diagnosis | Give lithium, ADHD, and metabolic residuals separate current-main reproducers and candidates. Behavior canaries require approval; stop any item without a deterministic reproducer or on regression. | -| 14 | `#029` | A2 | Specialist — answer quality/clinical safety | After `#051`/`#023` and `#018` | 0.5–1 day inventory; 1–3 days per fix | Re-enumerate current fallback stubs and fix one causal cluster at a time without weakening grounding/citation gates. Stop if a change merely makes the metric easier to pass. | -| 15 | `#001` | A2 | Specialist — retrieval/ranking | After `#051`/`#023` and rollout approval | 0.5–1 day plus canary | Keep semantic reranking off unless an approved ambiguity comparison preserves 36/36, recall 1.0, zero per-case regressions, and shows measured gain; otherwise record keep-off and stop. | -| 16 | `#025` | A2 | Operator — Railway/GitHub/chat/Supabase | Next approved observability window | 1–3 hours/channel | Choose owned deployment, CI, ingestion, and SLO alerts; mock first, then one approved controlled provider event/channel. The merged Supabase trigger remains inert until its verified inputs are configured. Stop without an accountable responder. | -| 17 | `#064` | A2 | High — frontend/browser | After higher-acuity local fixes; before the release UI gate | 4–8 hours | Preserve the isolated dirty formulation/contrast patch, reconcile its intent against current `main`, and run focused Playwright plus `verify:ui`. Stop rather than overwriting unrelated work or weakening access-control assertions. | -| 18 | `#055` | A2 | Specialist release owner + Operator | Before next full-confidence release/handoff | 2–4 hours plus runtime | On one exact SHA, run local/provider gates, Firefox/WebKit, required hosted CI, and close actionable GitHub threads. Stop at first failure and rerun only the repaired smallest gate. | -| 19 | `#056` | A2 | Operator — Supabase/Railway + Specialist | After cost/ownership approval | 0.5–1 day | Provision isolated `Clinical KB Staging` with synthetic data and distinct secrets. Verify identity, schema, indexing, health, and data boundary; never copy production clinical documents. | -| 20 | `#057` | A2 | High — release/SRE + Operator | After `#056` | 2–4 hours plus soak | Run documented staging soak and rollback against an exact candidate. Retain latency/error/rollback evidence; stop on unsafe data, identity mismatch, or unowned rollback. | -| 21 | `#058` | A2 | Operator — production data + Specialist | Next approved production verification window | 30–60 min read-only; 1–2 hours if needed | Verify registry/differentials/medications are non-empty before writing; seed only confirmed gaps idempotently. Stop when healthy or owner/project identity is ambiguous. | -| 22 | `#007` | A3 | Operator decision + Standard frontend | When product chooses canonical Tools route | 15–30 min decision; 0.5 day | Align navigation, redirect, sitemap, and reachability around one entry point. Stop while the standalone page has an unresolved requirement. | -| 23 | `#011` | A3 | Operator — Supabase capacity | Immediately before first compute scale-up | 30–60 min plus observation | Switch Auth to percentage allocation, record before/after, and run approved advisor/health checks. Stop if no scale-up is planned. | -| 24 | `#017` | A3 | High — performance/browser | Before `#012`/`#013`/`#016`; approved live-site window | 1–2 hours | Capture reproducible mobile/desktop Lighthouse/Web-Vitals evidence and decide whether payload work is justified. Stop if metrics are acceptable or evidence is too noisy. | -| 25 | `#024` | A3 | High — Next.js/Playwright/WebKit | After `#051`/`#023` or real Safari reproduction | 0.5–1 day | Distinguish test interception from a Safari defect. Apply test-only correction only with a discriminating repro; keep access-control assertions meaningful. | -| 26 | `#033` | A3 | Specialist — prompt/source governance | After `#022` and `#051`/`#023` | 1–2 days plus approved eval | Design unknown-vs-adverse metadata wording and prompt tests. Require no supported-grounding drop and zero citation failures; stop on broad over-caveating or degradation. | -| 27 | `#037` | A3 | Operator — clinical/product + Standard | Next trust-policy review | 30–60 min; up to 0.5 day | Decide whether routine claims cap at medium trust. Record policy; if accepted, change only the flag/expectations and run focused tests. | -| 28 | `#012`, `#013`, `#016` | A3 | High — bundling/runtime performance | After `#017` or equivalent evidence | 0.5–2 days/route | Optimize only a production route with measured payload/render/motion harm. Require material gain plus focused, `verify:cheap`, and browser evidence; stop on small gain. | -| 29 | `#035` | A3 | Specialist — evidence rules | After a demonstrated missed conflict | 0.5–1 day design; code separate | Define a clinically reviewed conflict class with positive and negative fixtures. Stop if no bounded class can be shown; behavior change requires protected review. | -| 30 | `#027` | Optional | Operator — SRE/provider | When an owned external alert path is wanted | 1–2 hours | Decide vendor/cost/privacy/owner; if accepted, prove one non-PHI outage and recovery alert. Stop when no responder owns it. | -| 31 | `#028` | Optional | Specialist privacy/observability + Operator | After privacy/ownership/cost approval | 1–3 days | Define vendor/region/retention/redaction/sampling/source-map envelope before SDK work. Prove no clinical text, identifiers, or secrets leave; stop if unacceptable. | -| 32 | `#038` | Optional | High — product/design architecture | When a new comparison surface is approved | 0.5–1 day | Define a shared interaction contract without flattening mode-specific content. Stop when no concrete new surface exists. | -| 33 | `#040` | Optional | High — visual QA/accessibility | When baseline owner/update workflow exists | 1–2 days | Establish a small stable desktop/mobile/accessibility baseline set. Do not make it blocking if flake or maintenance cost outweighs detection value. | -| 34 | `#039` | Optional | High — frontend architecture | During a concrete catalogue-toolbar project | 0.5–1 day inventory; 1–3 days code | Converge only repeated toolbar behavior without flattening search semantics. Stop when there is no bounded implementation target. | -| 35 | `#063` | A3 | High — product architecture + privacy | Only when the product owner wants to evaluate the feature | 0.5–1 day | Write a product/privacy/persistence brief for “Current Clinical Work” before storage or UI implementation. Stop if demand or safe persistence cannot be established. | -| 36 | `#065` | A2 | High — document-viewer UI | Only when the user explicitly resumes the paused task | 0.5–1.5 days | Finish the compact source-text accordion, citation/search auto-open, print restoration, and 320/390/1280 px coverage. Keep the preserved branch untouched until explicit resume; no provider calls. | - - +| Order | ID(s) | Acuity | Capability | When | Estimate | Outcome, gate, verification, and stopping condition | +| ----: | ---------------------- | -------- | ------------------------------------------- | ---------------------------------------------------------------------- | ---------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| 1 | `#059` | A1 | Operator security + independent reviewer | Immediate approved security window | 1–3 hours plus verification | Verify every reported exposed credential (GitHub, OpenAI, Supabase service role/database, E2E) is retired; rotate anything still valid and update only intended secret stores. Never record values; stop before provider action without approval. | +| 2 | `#053` | A1 | Operator — legal/privacy | Start now; finish before real patient use/privacy-approved release | 4–8 hours internal; 1–6 weeks elapsed | Execute DPAs; decide ZDR/residency; obtain cache behavior in writing; review subprocessors; obtain APP 8 and APP 5/1 counsel sign-off. Do not change public copy before approval. | +| 3 | `#067` | A3 | High — test reliability | Next flake-hardening window | 1–2 hours | Reproduce the load-sensitive reconciliation-preflight subprocess timeout, instrument its lifecycle, and make the smallest deterministic harness fix. Do not raise the global timeout or bypass the shared heavy-test lock without causal proof. | +| 4 | `#030` | A2 | High — evaluation semantics | Decision-ready | 2–4 hours | Require distinct source identities for distinct comparison slots. Run focused matching tests, typecheck, and `verify:cheap`; stop without changing strict aliases, retrieval, or ranking. | +| 5 | `#069` | A3 | Specialist — retrieval latency | After hosted apply of PR #1133 migrations; approval-gated live profile | 30–60 min | Operator applies `20260724120000`/`20260724130000`/`20260724130100`, then re-profiles `match_document_table_facts_text` (~70ms-class plans). Stop without mutating ranking or unpaid evals. Cloud agent blocked: no DB URL / MCP auth; live profile hit Unregistered API key. | 30–60 min | Re-profile `match_document_table_facts_text` with `explain_retrieval_rpc` / `profile:retrieval` after the plpgsql+EXECUTE migration is on the target project. Expect ~70ms-class plans; no ranking change. Stop without mutating ranking or calling unpaid evals. | +| 6 | `#019` | A2 | Specialist — RAG answer pipeline | Local reproducer now; behavior change after `#051`/`#023` | 0.5–1 day reproducer | Reproduce admission-source loss in the fallback layer using PR #1096’s source shape. Any behavior change needs protected review and an approved baseline/post canary; stop if independently non-reproducible. | +| 7 | `#054` | A2 | Standard locally; Operator hosted | Local safety identifier now; hosted next approved window | 15–30 min local; 1–2 hours hosted | Presence-check and fill confirmed secret/config gaps with distinct per-environment values. Never record values. Require clean readiness/secret checks; stop on ambiguous environment or project identity. | +| 8 | `#022` | A2 | Operator — clinical governance + Specialist | Decision-ready | 1–2 hours policy; 0.5–1 day first ten | Decide BMJ attestation policy and review the ten highest-impact local documents. Record reviewer/evidence/time; stop after ten and remeasure warning debt. | +| 9 | `#051`, `#023` | A2 | Specialist — RAG diagnostics | After scheduled 2026-07-26 run | 2–4 hours | With GitHub-read approval, compare structured canary/browser/irrelevant-at-10 artifacts without dispatching a rerun. Record deterministic/provider/latency deltas and disposition residuals; stop without spending. | +| 10 | `#018` | A2 | Specialist — clinical RAG/retrieval | After `#051`/`#023`, one mechanism at a time | 1–2 days diagnosis | Give lithium, ADHD, and metabolic residuals separate current-main reproducers and candidates. Behavior canaries require approval; stop any item without a deterministic reproducer or on regression. | +| 11 | `#029` | A2 | Specialist — answer quality/clinical safety | After `#051`/`#023` and `#018` | 0.5–1 day inventory; 1–3 days per fix | Re-enumerate current fallback stubs and fix one causal cluster at a time without weakening grounding/citation gates. Stop if a change merely makes the metric easier to pass. | +| 12 | `#001` | A2 | Specialist — retrieval/ranking | After `#051`/`#023` and rollout approval | 0.5–1 day plus canary | Keep semantic reranking off unless an approved ambiguity comparison preserves 36/36, recall 1.0, zero per-case regressions, and shows measured gain; otherwise record keep-off and stop. | +| 13 | `#025` | A2 | Operator — Railway/GitHub/chat/Supabase | Next approved observability window | 1–3 hours/channel | Choose owned deployment, CI, ingestion, and SLO alerts; mock first, then one approved controlled provider event/channel. The merged Supabase trigger remains inert until its verified inputs are configured. Stop without an accountable responder. | +| 14 | `#064` | A2 | High — frontend/browser | After higher-acuity local fixes; before the release UI gate | 4–8 hours | Preserve the isolated dirty formulation/contrast patch, reconcile its intent against current `main`, and run focused Playwright plus `verify:ui`. Stop rather than overwriting unrelated work or weakening access-control assertions. | +| 15 | `#055` | A2 | Specialist release owner + Operator | Before next full-confidence release/handoff | 2–4 hours plus runtime | On one exact SHA, run local/provider gates, Firefox/WebKit, required hosted CI, and close actionable GitHub threads. Stop at first failure and rerun only the repaired smallest gate. | +| 16 | `#056` | A2 | Operator — Supabase/Railway + Specialist | After cost/ownership approval | 0.5–1 day | Provision isolated `Clinical KB Staging` with synthetic data and distinct secrets. Verify identity, schema, indexing, health, and data boundary; never copy production clinical documents. | +| 17 | `#057` | A2 | High — release/SRE + Operator | After `#056` | 2–4 hours plus soak | Run documented staging soak and rollback against an exact candidate. Retain latency/error/rollback evidence; stop on unsafe data, identity mismatch, or unowned rollback. | +| 18 | `#058` | A2 | Operator — production data + Specialist | Next approved production verification window | 30–60 min read-only; 1–2 hours if needed | Verify registry/differentials/medications are non-empty before writing; seed only confirmed gaps idempotently. Stop when healthy or owner/project identity is ambiguous. | +| 19 | `#007` | A3 | Operator decision + Standard frontend | When product chooses canonical Tools route | 15–30 min decision; 0.5 day | Align navigation, redirect, sitemap, and reachability around one entry point. Stop while the standalone page has an unresolved requirement. | +| 20 | `#011` | A3 | Operator — Supabase capacity | Immediately before first compute scale-up | 30–60 min plus observation | Switch Auth to percentage allocation, record before/after, and run approved advisor/health checks. Stop if no scale-up is planned. | +| 21 | `#017` | A3 | High — performance/browser | Before `#012`/`#013`/`#016`; approved live-site window | 1–2 hours | Capture reproducible mobile/desktop Lighthouse/Web-Vitals evidence and decide whether payload work is justified. Stop if metrics are acceptable or evidence is too noisy. | +| 22 | `#024` | A3 | High — Next.js/Playwright/WebKit | After `#051`/`#023` or real Safari reproduction | 0.5–1 day | Distinguish test interception from a Safari defect. Apply test-only correction only with a discriminating repro; keep access-control assertions meaningful. | +| 23 | `#033` | A3 | Specialist — prompt/source governance | After `#022` and `#051`/`#023` | 1–2 days plus approved eval | Design unknown-vs-adverse metadata wording and prompt tests. Require no supported-grounding drop and zero citation failures; stop on broad over-caveating or degradation. | +| 24 | `#037` | A3 | Operator — clinical/product + Standard | Next trust-policy review | 30–60 min; up to 0.5 day | Decide whether routine claims cap at medium trust. Record policy; if accepted, change only the flag/expectations and run focused tests. | +| 25 | `#012`, `#013`, `#016` | A3 | High — bundling/runtime performance | After `#017` or equivalent evidence | 0.5–2 days/route | Optimize only a production route with measured payload/render/motion harm. Require material gain plus focused, `verify:cheap`, and browser evidence; stop on small gain. | +| 26 | `#035` | A3 | Specialist — evidence rules | After a demonstrated missed conflict | 0.5–1 day design; code separate | Define a clinically reviewed conflict class with positive and negative fixtures. Stop if no bounded class can be shown; behavior change requires protected review. | +| 27 | `#027` | Optional | Operator — SRE/provider | When an owned external alert path is wanted | 1–2 hours | Decide vendor/cost/privacy/owner; if accepted, prove one non-PHI outage and recovery alert. Stop when no responder owns it. | +| 28 | `#028` | Optional | Specialist privacy/observability + Operator | After privacy/ownership/cost approval | 1–3 days | Define vendor/region/retention/redaction/sampling/source-map envelope before SDK work. Prove no clinical text, identifiers, or secrets leave; stop if unacceptable. | +| 29 | `#038` | Optional | High — product/design architecture | When a new comparison surface is approved | 0.5–1 day | Define a shared interaction contract without flattening mode-specific content. Stop when no concrete new surface exists. | +| 30 | `#040` | Optional | High — visual QA/accessibility | When baseline owner/update workflow exists | 1–2 days | Establish a small stable desktop/mobile/accessibility baseline set. Do not make it blocking if flake or maintenance cost outweighs detection value. | +| 31 | `#039` | Optional | High — frontend architecture | During a concrete catalogue-toolbar project | 0.5–1 day inventory; 1–3 days code | Converge only repeated toolbar behavior without flattening search semantics. Stop when there is no bounded implementation target. | +| 32 | `#063` | A3 | High — product architecture + privacy | Only when the product owner wants to evaluate the feature | 0.5–1 day | Write a product/privacy/persistence brief for “Current Clinical Work” before storage or UI implementation. Stop if demand or safe persistence cannot be established. | +| 33 | `#065` | A2 | High — document-viewer UI | Only when the user explicitly resumes the paused task | 0.5–1.5 days | Finish the compact source-text accordion, citation/search auto-open, print restoration, and 320/390/1280 px coverage. Keep the preserved branch untouched until explicit resume; no provider calls. | + + ## Open items @@ -97,7 +95,6 @@ removed after current-main verification; it is not missing recommended work. | ID | Pri | Type | Summary | Detail / next action | Source | Added | | ---- | --- | ----- | --------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | ---------- | | #059 | P1 | task | Verify containment of every credential reported exposed in chat | **Outcome:** every reported exposed credential is rejected or retired. **Next:** in approved security windows, verify and revoke or rotate the GitHub token, OpenAI key, Supabase service-role JWT, database password, and E2E credential; create replacements only when required and update only intended secret stores. **Success:** provider evidence confirms the old credentials cannot authenticate, replacements are distinct and minimally scoped, presence/readiness checks pass, and secret scans remain clean. **Stop:** no provider or secret-store action without approval; never print or paste values into Git, logs, issues, or chat. | session 2026-07-24 security reconciliation; AI Agent Target Manifest | 2026-07-24 | -| #062 | P2 | issue | Upload crash can strand a queued document without a job | **Outcome:** a crash between document and job creation cannot strand an upload indefinitely. **Next:** add the stranded-row reproducer, then choose the smallest idempotent atomic-enqueue RPC or bounded scheduled sweep consistent with ownership and rollback contracts. **Success:** exactly one recoverable job is created, existing open jobs do not duplicate, and owner scope remains intact. **Verify:** focused upload/recovery/schema tests, migration guards, disposable replay if needed, `verify:cheap`, and production-readiness. **Stop:** hosted changes require approval; no at-least-once claim until the crash case passes. | `src/app/api/upload/route.ts`; `docs/webhooks.md` | 2026-07-24 | | #063 | P3 | rec | Define “Current Clinical Work” before implementation | **Outcome:** decide whether a workspace combining saved comparisons, partial formulation work, recent tools, and pinned source sets is worth building. **Next:** write a product/privacy/persistence brief only; do not build storage or UI. **Success:** define users, data classes, lifecycle, cross-device expectations, deletion, failure states, demand evidence, and the smallest testable slice. **Stop:** close the idea if demand or safe persistence cannot be established. | session 2026-07-24; `src/lib/tools-catalog.ts` | 2026-07-24 | | #064 | P2 | task | Reconcile the preserved browser and contrast patch | **Outcome:** the isolated dirty formulation/contrast patch is safely landed or explicitly dispositioned. **Next:** rebase its intent against current `main` without overwriting the worktree, then run focused Playwright coverage and `verify:ui`. **Success:** intended disabled/contrast behavior is accessible, browser assertions remain meaningful, and unrelated work is preserved. **Stop:** do not discard or auto-merge the dirty worktree; pause on ambiguous ownership or scope. | `agent/formulation-disabled-contrast`; session 2026-07-24 | 2026-07-24 | | #065 | P2 | task | Complete the paused compact document source-text accordion | **Outcome:** the document viewer uses compact nested disclosures while retaining complete text, citation/search navigation, print behavior, and composer clearance. **Next:** only when the user explicitly resumes, reconcile `codex/chat-document-text-accordion-7cb4` with current `main` and complete the focused 320/390/1280 px tests. **Success:** default disclosures are closed; deep links and search open only the active passage; printing expands/restores state; no overflow. **Verify:** focused document-viewer Playwright, `verify:cheap`, `verify:ui`, and static production-readiness. **Stop:** remain paused until explicit user return; no provider calls. | paused document-viewer task; `codex/chat-document-text-accordion-7cb4` | 2026-07-24 | @@ -105,13 +102,13 @@ removed after current-main verification; it is not missing recommended work. | #067 | P2 | issue | Reconciliation preflight test times out under full-suite load | **Outcome:** the reconciliation preflight subprocess test is deterministic under the repository's serialized heavy-test workflow. During PR #1119 validation, the 30-second test timeout occurred twice under loaded full-suite execution, while the isolated file passed 5/5 and a separate `verify:cheap` full suite passed. **Next:** reproduce with timing around subprocess startup, output and teardown, then fix the smallest proven harness lifecycle cause. **Success:** repeated focused and full-suite runs complete without extending the global timeout. **Stop:** do not hide the cause by raising broad timeouts, adding retries, or bypassing the shared test lock. | `tests/reconciliation-preflight.test.ts`; PR #1119 validation; session 2026-07-24 | 2026-07-24 | | #051 | P2 | task | Stabilise the live answer-quality canary before more RAG tuning | Diagnostics landed in PR #1095: structured JSON/Markdown artifacts now record the actual checked-out SHA, run identity and latency context, and the offline trend tool separates content, provider-route and latency outcomes. First validating run `30018289898` recorded the expected tree and cost, with 36/36 retrieval green, but one report cannot establish variability; PR #1097 prevents a single failure being mislabeled as repeated. Next: compare the scheduled 2026-07-26 structured report with this run. Do not spend on an immediate retry or reapply the archived lithium guard before that comparison. | PR #1095; run `30018289898`; PR #1097; archive ref `refs/archive/rejected-rag/20260723/monitoring-subject-gate` | 2026-07-23 | | #001 | P2 | task | Semantic reranking still gated off | `RAG_SEMANTIC_RERANK_ENABLED=false` from PR #901. Do not enable until the provider-backed 36/36 retrieval-quality gate **and** an ambiguity-focused canary are explicitly approved and recorded. | `docs/process-hardening.md` (Semantic reranking rollout debt); PR #901 | 2026-07-21 | -| #052 | P2 | issue | Reindex can overlap a fresh agent-enrichment pass | Full/retry single and bulk routes consult `ingestion_jobs` but not the implemented `hasActiveAgentEnrichmentJob` predicate. Add red route tests, then block fresh `indexing_v3_agent_jobs.status='processing'` leases before mutation while keeping stale leases and enrichment mode unchanged. | `src/lib/ingestion-mutation-safety.ts`; single/bulk reindex routes; repository audit 2026-07-24 | 2026-07-24 | +| #069 | P3 | task | Live-profile table-facts plpgsql+EXECUTE latency | Migration `20260724120000_table_facts_plpgsql_execute.sql` plus P3 follow-ups (`20260724130000_*`, `20260724130100_*`) are in PR #1133. Hosted apply is blocked in this environment (no `SUPABASE_DB_URL`; Supabase MCP `needsAuth`). A live `profile:retrieval --rpc match_document_table_facts_text --analyze` attempt returned `Unregistered API key` against the injected service-role secret. **Next:** operator applies the three pending migrations on the live target project, then re-runs approval-gated `profile:retrieval` / `explain_retrieval_rpc` and confirms ~70ms-class plans with no ranking change. **Stop** without mutating ranking. | session 2026-07-24 Database interface audit; PR #1133 | 2026-07-24 | | #053 | P1 | task | Execute cross-border privacy/legal package | Execute OpenAI and Railway DPAs; decide ZDR and Australian data residency; obtain prompt-cache behavior in writing; review subprocessors; obtain APP 8 and APP 5/1 counsel sign-off. Do not represent the release as privacy-approved or alter final public privacy wording before sign-off. | `docs/openai-cross-border-basis.md`; `docs/privacy-impact-assessment.md` | 2026-07-24 | | #054 | P2 | task | Reconcile local and hosted secrets/config | Presence-check safety-identifier, query-hash, deep-probe, Supabase service-role, OpenAI, project-identity, and schedule settings; set only confirmed gaps with distinct per-environment values. Never record secret values. Provider reads/writes require approval. | `.env.example`; production-readiness warning; `docs/operator-backlog.md` | 2026-07-24 | | #055 | P2 | task | Run one exact-SHA full release and PR gate | Before the next full-confidence release/handoff, record the candidate/PR SHA and run the local/provider release gates, Firefox/WebKit, required hosted CI, and actionable GitHub review-thread closure once. Stop at the first actionable failure and rerun only the repaired smallest gate. | `docs/launch-operator-runbook.md`; `docs/codex-review-protocol.md` | 2026-07-24 | | #056 | P2 | task | Provision isolated staging environment | After explicit cost/ownership approval, provision `Clinical KB Staging` Supabase and Railway tiers with distinct secrets and synthetic/non-clinical data. Verify identity, schema, indexing, health, and the production-data boundary. | `docs/staging-setup.md`; `docs/operator-backlog.md` | 2026-07-24 | | #057 | P2 | task | Complete staging soak and rollback rehearsal | After #056, run the documented soak and rollback against an exact candidate; retain latency/error/rollback evidence. Stop on unsafe data, identity mismatch, or an unowned rollback decision. | `docs/launch-operator-runbook.md`; `docs/capacity-review.md` | 2026-07-24 | -| #058 | P2 | task | Verify production content before any seed write | Against `Clinical KB Database`, verify registry, differentials, and medications surfaces are non-empty before writing. Seed only confirmed gaps idempotently with approved owner/project identity and confirmation flags. | `docs/launch-operator-runbook.md`; `docs/operator-backlog.md` | 2026-07-24 | +| #058 | P2 | task | Verify production content before any seed write | Against `the live target project`, verify registry, differentials, and medications surfaces are non-empty before writing. Seed only confirmed gaps idempotently with approved owner/project identity and confirmation flags. | `docs/launch-operator-runbook.md`; `docs/operator-backlog.md` | 2026-07-24 | | #005 | P3 | rec | `finalScore` saturates at clamp ceiling | Base + ~40 stacked boosts routinely exceed 1.0, so strong matches tie at 1.0 and order by an arbitrary `document_id` tiebreak. If ranking is ever revisited, break ties by the **pre-clamp** score rather than raising the `[0,1]` ceiling (downstream gates assume `[0,1]`). Ordering already sorts by the unbounded pre-clamp `rankScore` (`clinical-search.ts:1735,1927,1950-1955`), so the clamp confines only the reported confidence value, not result order. Not a defect on the current golden set; any change here is a protected RAG surface (canary required). | `docs/rag-hybrid-findings-and-todo.md` P1 item 4; `src/lib/clinical-search.ts:1735` | 2026-07-21 | | #007 | P3 | rec | `/tools` vs `/?mode=tools` parallel Tools entry points | `/tools` (standalone `ApplicationsLauncherPage`) has no inbound in-app link; the sidebar Tools item uses `/?mode=tools`. Decide the canonical entry point and wire nav consistently, or drop the standalone `/tools` page + `/applications` redirect. Currently allowlisted in `tests/route-reachability.test.ts`. | `src/app/tools/page.tsx`; `src/app/applications/route.ts` | 2026-07-21 | | #009 | P3 | rec | Confirm `/api/jobs` is intentionally server/ops-only | No client `fetch()` reaches `/api/jobs` (only tests import it). Confirm it is a deliberate ops/manual surface; if abandoned, remove it. | `src/app/api/jobs/route.ts` | 2026-07-21 | @@ -148,6 +145,9 @@ Move resolved rows here with the resolution date and a one-line outcome. Keep th | ID | Type | Summary | Outcome | Resolved | | ---- | ----- | ---------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | +| #068 | task | Regenerate full drift-manifest snapshot after schema hygiene | Full Docker `npm run drift:manifest` replay succeeded on a Docker-capable host; `supabase/drift-manifest.json` now carries live `def_hash` values for the plpgsql table-facts body (offline generator_note removed). | 2026-07-24 | +| #052 | issue | Reindex can overlap a fresh agent-enrichment pass | Full/retry single and bulk reindex now preflight `listDocumentsWithActiveAgentEnrichment` / `hasActiveAgentEnrichmentJob`; enrichment mode unchanged; stale leases still do not block. Focused lease-gate tests plus private-access reindex coverage prove the 409 path. | 2026-07-24 | +| #062 | issue | Upload crash can strand a queued document without a job | Added stranded-row reproducer and owner-scoped aged `queued`-without-open-job recovery via `recover:ingestion --include-stranded-queued`, reusing idempotent `enqueueDocumentReindexJob`. Hosted apply of any future atomic-enqueue RPC remains optional; sweep is the smallest recovery path. | 2026-07-24 | | #060 | issue | Safety Plan Generator contradicted the privacy contract | PR #1119 removed patient identifier entry, leaves the post-export name line blank, and aligned tool, privacy and PIA copy. DOM/privacy tests and Chromium copy/print/network coverage prove working content remains in React memory with no fetch/XHR; hosted Production UI, build, unit, policy, safety, static-analysis and secret checks passed. Support-contact details remain classified as sensitive local-only working content. | 2026-07-24 | | #061 | issue | Missing answer relevance metadata was treated as source-backed | PR #1125 now requires explicit source-backed relevance for trusted/grounded presentation and prevents visual tables, clinical-note sections and quotes, and comparison metadata from bypassing the render model. Three actionable P2 review paths were fixed; focused policy/DOM tests, offline RAG, production-readiness, build, unit, static, security, and Production UI gates passed. No retrieval, ranking, generation, provider, or data behavior changed. | 2026-07-24 | | #034 | issue | Answer cache can serve stale governance metadata | Current-source verification found direct route coverage already asserts RAG-cache invalidation on document PATCH, source review, label, bulk, and reindex mutation paths. The residual test recommendation is already met; changing the protected cache key is unnecessary. | 2026-07-24 | diff --git a/docs/reindex-runbook.md b/docs/reindex-runbook.md index 3469965dc2..ef308c364d 100644 --- a/docs/reindex-runbook.md +++ b/docs/reindex-runbook.md @@ -56,6 +56,7 @@ If the response includes `safety.safeToRun: false`, do not retry repeatedly. Use - `supabase_unavailable`: pause reindexing and rerun `npm run supabase:recovery-status`. - `active_jobs`: wait for pending or processing jobs to finish before retrying. - `stale_processing_jobs`: run `npm run recover:ingestion -- --apply --limit 20`, then rerun `npm run reindex:health`. +- `queued` documents with no open job (upload crash between document insert and job insert): dry-run `npm run recover:ingestion -- --include-stranded-queued`, then re-run with `--apply` after reviewing the candidate list. Optional `--owner-id` and `--stranded-min-age-minutes` keep the sweep bounded. The preflight response also includes active job counts and job metadata for operator review. diff --git a/docs/scale-readiness-review.md b/docs/scale-readiness-review.md index 1beff40330..80cfe31929 100644 --- a/docs/scale-readiness-review.md +++ b/docs/scale-readiness-review.md @@ -106,13 +106,12 @@ non-inlined hybrid RPCs that showed the same signature (memory-cards 687 ms, index-units 444 ms, embedding-fields 520 ms) — same generic-plan cause, same one-line fix. -**Full fix (recovers 5.3 s → ~70 ms, ≈75×, still results-identical, larger -change):** convert the function to `LANGUAGE plpgsql` and run the body via -`RETURN QUERY EXECUTE '' USING …`. Dynamic `EXECUTE` re-plans per call -with the actual bound values (a one-shot custom plan), sidestepping the cached -generic plan entirely. Worth doing given the answer path calls this RPC up to -three times in parallel (`rag.ts`), but it needs the drifted body reconciled to -live first. +**Full fix (recovers 5.3 s → ~70 ms, ≈75×, still results-identical):** convert +the function to `LANGUAGE plpgsql` and run the body via +`RETURN QUERY EXECUTE … USING …`. Shipped in migration +`20260724120000_table_facts_plpgsql_execute.sql` (mirrored in `schema.sql`). +Dynamic `EXECUTE` re-plans per call with the actual bound values. Re-profile +live after hosted apply (`#069`); ranking/result predicates are unchanged. Not runnable from the review environment: applying either fix needs the linked `supabase` CLI (operator credentials) — `db push`/`link` are not authenticated diff --git a/docs/webhooks.md b/docs/webhooks.md index 29be14f568..59b448d20a 100644 --- a/docs/webhooks.md +++ b/docs/webhooks.md @@ -189,7 +189,10 @@ missing or the POST fails (it just does nothing, mirroring the receiver's inert > an orphan, but a process crash/timeout/kill _between_ the two inserts can still > strand the same `queued`-without-job row nothing sweeps. The only true > must-never-drop guarantees are an atomic enqueue (document + job in one -> transaction/RPC) or a scheduled queued-without-job sweep. This trigger is a +> transaction/RPC) or a scheduled queued-without-job sweep. Operator recovery +> for aged stranded rows is available via +> `npm run recover:ingestion -- --include-stranded-queued` (optional +> `--owner-id`, `--stranded-min-age-minutes`, `--apply`). This trigger is a > low-latency optimisation, not a delivery guarantee. ```sql diff --git a/scripts/recover-ingestion-queue.ts b/scripts/recover-ingestion-queue.ts index c74817fa44..f1f25308a2 100644 --- a/scripts/recover-ingestion-queue.ts +++ b/scripts/recover-ingestion-queue.ts @@ -35,8 +35,8 @@ function supabaseStageError(stage: string, error: unknown) { return wrapped; } -const booleanFlags = new Set(["--apply", "--yes"]); -const valueFlags = new Set(["--stale-after-minutes", "--limit"]); +const booleanFlags = new Set(["--apply", "--yes", "--include-stranded-queued"]); +const valueFlags = new Set(["--stale-after-minutes", "--limit", "--stranded-min-age-minutes", "--owner-id"]); // Audit L2 (hardened after diff review): this script mutates ingestion state, // so argument parsing fails loudly on ANY surprise — @@ -71,11 +71,18 @@ function parseArgs(argv: string[]) { } return parsed; }; + const ownerId = values.get("--owner-id"); + if (ownerId !== undefined && ownerId.trim().length === 0) { + throw new Error("--owner-id must be a non-empty UUID when provided."); + } return { apply: booleans.has("--apply"), yes: booleans.has("--yes"), + includeStrandedQueued: booleans.has("--include-stranded-queued"), staleAfterMinutes: positiveIntFor("stale-after-minutes"), + strandedMinAgeMinutes: positiveIntFor("stranded-min-age-minutes"), limit: positiveIntFor("limit"), + ownerId: ownerId?.trim(), }; } @@ -133,13 +140,34 @@ async function main() { console.log(`Remaining (over limit): ${remainingCount}`); } - if (actions.length === 0) { + let stranded: Awaited< + ReturnType<(typeof import("@/lib/stranded-queued-recovery"))["listStrandedQueuedDocuments"]> + > | null = null; + if (args.includeStrandedQueued) { + const { listStrandedQueuedDocuments, STRANDED_QUEUED_DEFAULT_MIN_AGE_MINUTES } = + await import("@/lib/stranded-queued-recovery"); + const strandedMinAgeMinutes = args.strandedMinAgeMinutes ?? STRANDED_QUEUED_DEFAULT_MIN_AGE_MINUTES; + stranded = await listStrandedQueuedDocuments({ + supabase, + minAgeMinutes: strandedMinAgeMinutes, + limit, + ownerId: args.ownerId ?? null, + }); + console.log("\n=== Stranded queued-without-job recovery ==="); + console.log(`Min age : ${strandedMinAgeMinutes} min`); + console.log(`Owner scope : ${args.ownerId ?? "(all owners)"}`); + console.log(`Stranded candidates : ${stranded.length}`); + for (const document of stranded) { + console.log(` - ${document.id} owner=${document.owner_id ?? "null"} updated_at=${document.updated_at}`); + } + } + + if (actions.length === 0 && (stranded?.length ?? 0) === 0) { console.log("\nNothing to recover. Queue looks healthy."); return; } let shouldApply = args.apply; - if (!shouldApply) { if (args.yes) { shouldApply = true; @@ -154,56 +182,75 @@ async function main() { return; } - console.log("\nApplying recovery..."); + if (actions.length > 0) { + console.log("\nApplying job recovery..."); - for (const documentId of resetDocumentIds) { - const { error: resetError } = await supabase.rpc("reset_document_index", { p_document_id: documentId }); - if (resetError) throw supabaseStageError("reset document index", resetError); - const { error: documentError } = await supabase - .from("documents") - .update({ status: "queued", error_message: null, page_count: 0, chunk_count: 0, image_count: 0 }) - .eq("id", documentId); - if (documentError) throw supabaseStageError("reset document status", documentError); - } + for (const documentId of resetDocumentIds) { + const { error: resetError } = await supabase.rpc("reset_document_index", { p_document_id: documentId }); + if (resetError) throw supabaseStageError("reset document index", resetError); + const { error: documentError } = await supabase + .from("documents") + .update({ status: "queued", error_message: null, page_count: 0, chunk_count: 0, image_count: 0 }) + .eq("id", documentId); + if (documentError) throw supabaseStageError("reset document status", documentError); + } + + for (const action of actions) { + if (action.action === "supersede") { + const { error: supersedeError } = await supabase + .from("ingestion_jobs") + .update({ + status: "completed", + stage: "superseded by successful index", + progress: 100, + error_message: null, + locked_at: null, + locked_by: null, + completed_at: new Date().toISOString(), + }) + .eq("id", action.jobId); + if (supersedeError) throw supabaseStageError("supersede sibling ingestion job", supersedeError); + continue; + } - for (const action of actions) { - if (action.action === "supersede") { - const { error: supersedeError } = await supabase + const { error: retryError } = await supabase .from("ingestion_jobs") .update({ - status: "completed", - stage: "superseded by successful index", - progress: 100, + status: "pending", + stage: "queued after recovery", + progress: 0, + attempt_count: 0, error_message: null, locked_at: null, locked_by: null, - completed_at: new Date().toISOString(), + next_run_at: new Date().toISOString(), + completed_at: null, }) .eq("id", action.jobId); - if (supersedeError) throw supabaseStageError("supersede sibling ingestion job", supersedeError); - continue; + if (retryError) throw supabaseStageError("requeue ingestion job", retryError); } - const { error: retryError } = await supabase - .from("ingestion_jobs") - .update({ - status: "pending", - stage: "queued after recovery", - progress: 0, - attempt_count: 0, - error_message: null, - locked_at: null, - locked_by: null, - next_run_at: new Date().toISOString(), - completed_at: null, - }) - .eq("id", action.jobId); - if (retryError) throw supabaseStageError("requeue ingestion job", retryError); + console.log("Ingestion queue recovery applied."); + if (remainingCount > 0) { + console.log(`\n${remainingCount} action(s) remain over the limit. Re-run to process the next batch.`); + } } - console.log("Ingestion queue recovery applied."); - if (remainingCount > 0) { - console.log(`\n${remainingCount} action(s) remain over the limit. Re-run to process the next batch.`); + if (stranded && stranded.length > 0) { + const { recoverStrandedQueuedDocuments } = await import("@/lib/stranded-queued-recovery"); + const strandedResults = await recoverStrandedQueuedDocuments({ supabase, documents: stranded }); + const enqueued = strandedResults.filter((result) => result.outcome === "enqueued").length; + const alreadyActive = strandedResults.filter((result) => result.outcome === "already_active").length; + const errors = strandedResults.filter((result) => result.outcome === "error"); + console.log(`\nStranded enqueued : ${enqueued}`); + console.log(`Stranded already active : ${alreadyActive}`); + if (errors.length > 0) { + console.log(`Stranded errors : ${errors.length}`); + for (const error of errors) { + console.log(` - ${error.documentId}: ${error.message}`); + } + throw new Error(`Stranded queued recovery reported ${errors.length} error(s).`); + } } } diff --git a/src/app/api/documents/[id]/reindex/route.ts b/src/app/api/documents/[id]/reindex/route.ts index 6dc4993346..e9ee68cd28 100644 --- a/src/app/api/documents/[id]/reindex/route.ts +++ b/src/app/api/documents/[id]/reindex/route.ts @@ -6,6 +6,7 @@ import { activeIngestionJobColumns, buildActiveJobsSafetyResult, checkIngestionMutationSafety, + hasActiveAgentEnrichmentJob, ingestionMutationSafetyPayload, ingestionRollbackFenceStamp, type IngestionJobRow, @@ -67,6 +68,20 @@ export async function POST(request: Request, { params }: { params: Promise<{ id: }); if (!safety.ok) return NextResponse.json(ingestionMutationSafetyPayload(safety), { status: safety.status }); + // Full reindex deletes/rebuilds the same enrichment artifact families the + // agent writes. Block only while a fresh agent lease is live; enrichment + // mode keeps its own RPC concurrency (`request_indexing_v3_enrichment`). + if (mode !== "enrichment") { + const enrichmentActive = await hasActiveAgentEnrichmentJob({ + supabase, + documentId: id, + staleAfterMinutes: env.WORKER_STALE_AFTER_MINUTES, + }); + if (enrichmentActive) { + return NextResponse.json({ error: "Reindex is paused while enrichment is active." }, { status: 409 }); + } + } + if (mode === "enrichment") { const { data: queued, error: queueError } = await supabase.rpc("request_indexing_v3_enrichment", { p_document_id: id, diff --git a/src/app/api/documents/bulk/reindex/route.ts b/src/app/api/documents/bulk/reindex/route.ts index aa07e4810b..8ccc26002e 100644 --- a/src/app/api/documents/bulk/reindex/route.ts +++ b/src/app/api/documents/bulk/reindex/route.ts @@ -8,6 +8,7 @@ import { checkIngestionMutationSafety, ingestionMutationSafetyPayload, ingestionRollbackFenceStamp, + listDocumentsWithActiveAgentEnrichment, type IngestionJobRow, } from "@/lib/ingestion-mutation-safety"; import { consumeApiRateLimit, rateLimitJsonResponse } from "@/lib/api-rate-limit"; @@ -55,6 +56,34 @@ export async function POST(request: Request) { }); if (!safety.ok) return NextResponse.json(ingestionMutationSafetyPayload(safety), { status: safety.status }); + // Preflight conflict (same all-or-nothing shape as active ingestion jobs): + // full/retry rebuilds clash with a live agent enrichment pass. Enrichment + // mode is unchanged and uses its own RPC guard. For retry_failed, only the + // documents this mode will actually process (status=failed) are checked — + // leased non-failed selections are skipped later and must not 409 the batch. + if (parsed.mode !== "enrichment") { + const enrichmentCheckIds = + parsed.mode === "retry_failed" + ? documents.filter((document) => document.status === "failed").map((document) => document.id) + : documents.map((document) => document.id); + if (enrichmentCheckIds.length > 0) { + const blockedDocumentIds = await listDocumentsWithActiveAgentEnrichment({ + supabase, + documentIds: enrichmentCheckIds, + staleAfterMinutes: env.WORKER_STALE_AFTER_MINUTES, + }); + if (blockedDocumentIds.length > 0) { + return NextResponse.json( + { + error: "Bulk reindex is paused while enrichment is active for one or more selected documents.", + blockedDocumentIds, + }, + { status: 409 }, + ); + } + } + } + const results: Array<{ documentId: string; mode: string; ok: boolean; jobId?: string; error?: string }> = []; for (const document of documents) { diff --git a/src/lib/ingestion-mutation-safety.ts b/src/lib/ingestion-mutation-safety.ts index 365ac6c062..2cc7281a58 100644 --- a/src/lib/ingestion-mutation-safety.ts +++ b/src/lib/ingestion-mutation-safety.ts @@ -140,6 +140,28 @@ export async function hasActiveAgentEnrichmentJob(args: { staleAfterMinutes: number; now?: Date; }): Promise { + const blocked = await listDocumentsWithActiveAgentEnrichment({ + supabase: args.supabase, + documentIds: [args.documentId], + staleAfterMinutes: args.staleAfterMinutes, + now: args.now, + }); + return blocked.length > 0; +} + +// Batch form used by bulk reindex preflight so one query covers the selection +// set instead of N round-trips. Fresh `processing` leases block; stale leases +// and non-processing statuses do not (same predicate as the single-document +// helper). +export async function listDocumentsWithActiveAgentEnrichment(args: { + supabase: SupabaseAdminClient; + documentIds: string[]; + staleAfterMinutes: number; + now?: Date; +}): Promise { + const uniqueDocumentIds = Array.from(new Set(args.documentIds.filter(Boolean))); + if (uniqueDocumentIds.length === 0) return []; + // indexing_v3_agent_jobs is not in the generated Database types (it is a // worker-state table added by migration), so query it through an untyped // client the same way the reindex route paginates dynamic tables. @@ -147,14 +169,18 @@ export async function hasActiveAgentEnrichmentJob(args: { const { data, error } = await client .from("indexing_v3_agent_jobs") .select("document_id,status,locked_at,updated_at") - .eq("document_id", args.documentId) - .eq("status", "processing") - .limit(1); + .in("document_id", uniqueDocumentIds) + .eq("status", "processing"); if (error) throw new Error(error.message); const nowMs = (args.now ?? new Date()).getTime(); - return ((data ?? []) as AgentEnrichmentJobRow[]).some((job) => - isActiveAgentEnrichmentJob(job, args.staleAfterMinutes, nowMs), + return Array.from( + new Set( + ((data ?? []) as AgentEnrichmentJobRow[]) + .filter((job) => isActiveAgentEnrichmentJob(job, args.staleAfterMinutes, nowMs)) + .map((job) => job.document_id) + .filter((id): id is string => Boolean(id)), + ), ); } diff --git a/src/lib/stranded-queued-recovery.ts b/src/lib/stranded-queued-recovery.ts new file mode 100644 index 0000000000..5d202e4c17 --- /dev/null +++ b/src/lib/stranded-queued-recovery.ts @@ -0,0 +1,158 @@ +import "server-only"; +import { enqueueDocumentReindexJob, type EnqueueableDocument } from "@/lib/ingestion-enqueue"; +import type { createAdminClient } from "@/lib/supabase/admin"; + +type AdminClient = ReturnType; + +export type StrandedQueuedDocument = EnqueueableDocument & { + created_at: string; + updated_at: string; +}; + +export type StrandedQueuedRecoveryResult = + | { documentId: string; outcome: "enqueued"; jobId?: string } + | { documentId: string; outcome: "already_active" } + | { documentId: string; outcome: "document_deleted" } + | { documentId: string; outcome: "skipped_young"; ageMinutes: number } + | { documentId: string; outcome: "error"; message: string }; + +export const STRANDED_QUEUED_DEFAULT_MIN_AGE_MINUTES = 15; +export const STRANDED_QUEUED_DEFAULT_LIMIT = 20; +/** Hard cap on aged-queued rows scanned while hunting for stranded ones. */ +export const STRANDED_QUEUED_MAX_SCAN = 200; + +function ageMinutes(iso: string, nowMs: number) { + const time = Date.parse(iso); + if (!Number.isFinite(time)) return null; + return (nowMs - time) / 60_000; +} + +// Pure predicate used by the reproducer and recovery planner: a document is +// stranded when it is still `queued`, old enough that a mid-upload crash is +// plausible, and has no open pending/processing ingestion job. +export function isStrandedQueuedDocument(args: { + document: { status: string | null; created_at: string; updated_at?: string | null }; + openJobCount: number; + minAgeMinutes: number; + now?: Date; +}): boolean { + if (args.document.status !== "queued") return false; + if (args.openJobCount > 0) return false; + const nowMs = (args.now ?? new Date()).getTime(); + const age = ageMinutes(args.document.updated_at || args.document.created_at, nowMs); + if (age === null) return false; + return age >= args.minAgeMinutes; +} + +export async function listStrandedQueuedDocuments(args: { + supabase: AdminClient; + minAgeMinutes?: number; + limit?: number; + ownerId?: string | null; + now?: Date; +}): Promise { + const minAgeMinutes = args.minAgeMinutes ?? STRANDED_QUEUED_DEFAULT_MIN_AGE_MINUTES; + const limit = args.limit ?? STRANDED_QUEUED_DEFAULT_LIMIT; + const now = args.now ?? new Date(); + const cutoff = new Date(now.getTime() - minAgeMinutes * 60_000).toISOString(); + // Page past aged queued rows that already have open jobs so a busy queue + // cannot starve genuinely stranded documents behind the first `limit` rows. + const pageSize = Math.max(limit, STRANDED_QUEUED_DEFAULT_LIMIT); + // Always allow paging up to the hard scan cap so a busy prefix of open-job + // rows cannot exhaust the budget before `limit` stranded rows are collected. + const maxScan = STRANDED_QUEUED_MAX_SCAN; + + const stranded: StrandedQueuedDocument[] = []; + let offset = 0; + + while (stranded.length < limit && offset < maxScan) { + const fetchCount = Math.min(pageSize, maxScan - offset); + let query = args.supabase + .from("documents") + .select( + "id,owner_id,status,error_message,page_count,chunk_count,image_count,import_batch_id,created_at,updated_at", + ) + .eq("status", "queued") + .lt("updated_at", cutoff) + .order("updated_at", { ascending: true }); + + if (args.ownerId) { + query = query.eq("owner_id", args.ownerId); + } + + const { data, error } = await query.range(offset, offset + fetchCount - 1); + if (error) throw new Error(error.message); + + const candidates = (data ?? []) as StrandedQueuedDocument[]; + if (candidates.length === 0) break; + + const documentIds = candidates.map((document) => document.id); + const { data: openJobs, error: openJobsError } = await args.supabase + .from("ingestion_jobs") + .select("document_id") + .in("document_id", documentIds) + .in("status", ["pending", "processing"]); + if (openJobsError) throw new Error(openJobsError.message); + + const openJobCounts = new Map(); + for (const job of openJobs ?? []) { + const documentId = job.document_id; + if (!documentId) continue; + openJobCounts.set(documentId, (openJobCounts.get(documentId) ?? 0) + 1); + } + + for (const document of candidates) { + if (stranded.length >= limit) break; + if ( + isStrandedQueuedDocument({ + document: { + status: document.status ?? null, + created_at: document.created_at, + updated_at: document.updated_at, + }, + openJobCount: openJobCounts.get(document.id) ?? 0, + minAgeMinutes, + now, + }) + ) { + stranded.push(document); + } + } + + offset += candidates.length; + if (candidates.length < fetchCount) break; + } + + return stranded; +} + +export async function recoverStrandedQueuedDocuments(args: { + supabase: AdminClient; + documents: StrandedQueuedDocument[]; +}): Promise { + const results: StrandedQueuedRecoveryResult[] = []; + + for (const document of args.documents) { + try { + const result = await enqueueDocumentReindexJob({ + supabase: args.supabase, + document, + }); + if (result.outcome === "enqueued") { + const jobId = + result.job && typeof result.job === "object" && "id" in result.job ? String(result.job.id ?? "") : undefined; + results.push({ documentId: document.id, outcome: "enqueued", jobId: jobId || undefined }); + continue; + } + results.push({ documentId: document.id, outcome: result.outcome }); + } catch (error) { + results.push({ + documentId: document.id, + outcome: "error", + message: error instanceof Error ? error.message : String(error), + }); + } + } + + return results; +} diff --git a/supabase/drift-manifest.json b/supabase/drift-manifest.json index b515a88453..4f47aad295 100644 --- a/supabase/drift-manifest.json +++ b/supabase/drift-manifest.json @@ -1,9 +1,9 @@ { - "generated_at": "2026-07-23T23:41:25.333Z", + "generated_at": "2026-07-24T05:47:11.755Z", "generator": "scripts/generate-drift-manifest.ts", "postgres_image": "supabase/postgres:17.6.1.127", - "schema_sha256": "b00b143ba74e98525ba17f032245c05a7befff9adbc07c612f0f8de29855bc51", - "replay_seconds": 16, + "schema_sha256": "81fd99d0849bde3c68a7a476e923034e0bd0258fda08e8e67f35b1c426f110e4", + "replay_seconds": 17, "snapshot": { "views": [ { @@ -6890,7 +6890,7 @@ "postgres=X/postgres", "service_role=X/postgres" ], - "def_hash": "5028a7406b7014ea8e274d29c8bbf6f9", + "def_hash": "4a8d38962265b8fde233dc8657773bf1", "signature": "public.invoke_ingestion_worker(integer)" }, { @@ -7090,7 +7090,7 @@ "postgres=X/postgres", "service_role=X/postgres" ], - "def_hash": "b8569b910ab5fb3b6b23af78eeeed374", + "def_hash": "f68e03ca96f8403d171509a59a769682", "signature": "public.match_document_table_facts_text(text,integer,uuid[],uuid)" }, { diff --git a/supabase/migrations/20260724120000_table_facts_plpgsql_execute.sql b/supabase/migrations/20260724120000_table_facts_plpgsql_execute.sql new file mode 100644 index 0000000000..1168cb50cb --- /dev/null +++ b/supabase/migrations/20260724120000_table_facts_plpgsql_execute.sql @@ -0,0 +1,167 @@ +-- Byte-identical result set for match_document_table_facts_text, but plan with +-- the actual bound parameter values each call. LANGUAGE sql + SET search_path +-- made the body non-inlinable, so Postgres cached a generic plan that could not +-- estimate trigram/tsv selectivity for unknown $1 and paid ~1–5s instead of +-- ~70ms (see docs/scale-readiness-review.md F1). Dynamic EXECUTE forces a +-- one-shot custom plan. RAG impact: no retrieval behaviour change — result rows +-- and ordering predicates are unchanged. + +create or replace function public.match_document_table_facts_text( + query_text text, + match_count integer default 16, + document_filters uuid[] default null, + owner_filter uuid default null +) +returns table ( + id uuid, + document_id uuid, + source_chunk_id uuid, + source_image_id uuid, + page_number integer, + table_title text, + row_label text, + clinical_parameter text, + threshold_value text, + action text, + text_rank double precision, + match_reason text +) +language plpgsql +stable +set search_path = public, extensions, pg_temp +set plan_cache_mode = 'force_custom_plan' +as $function$ +begin + return query execute $body$ + with query as ( + select + websearch_to_tsquery('english', coalesce($1, '')) as tsq, + lower(trim(regexp_replace(coalesce($1, ''), '\s+', ' ', 'g'))) as normalized, + string_to_array(lower(trim(regexp_replace(coalesce($1, ''), '\s+', ' ', 'g'))), ' ')::text[] as tokens + ), + doc_scope as ( + select d.id, d.metadata + from public.documents d + where d.status = 'indexed' + and public.retrieval_owner_matches($4, d.owner_id) + and ($3 is null or d.id = any($3)) + ), + fts_matches as ( + select + f.id, + f.document_id, + f.source_chunk_id, + f.source_image_id, + f.page_number, + f.table_title, + f.row_label, + f.clinical_parameter, + f.threshold_value, + f.action, + ts_rank_cd(f.search_tsv, q.tsq)::double precision as text_rank, + case + when coalesce(f.threshold_value, '') <> '' then 'table_threshold' + when coalesce(f.action, '') <> '' then 'table_action' + else 'table_row' + end as match_reason + from query q + join public.document_table_facts f on f.search_tsv @@ q.tsq + join doc_scope ds on ds.id = f.document_id + where public.is_committed_artifact_generation(f.metadata, ds.metadata) + order by ts_rank_cd(f.search_tsv, q.tsq) desc + limit greatest($2 * 5, 64) + ), + term_matches as ( + select + f.id, + f.document_id, + f.source_chunk_id, + f.source_image_id, + f.page_number, + f.table_title, + f.row_label, + f.clinical_parameter, + f.threshold_value, + f.action, + 0.45::double precision as text_rank, + 'term_overlap'::text as match_reason + from query q + join public.document_table_facts f + on cardinality(q.tokens) > 0 + and f.normalized_terms && q.tokens + join doc_scope ds on ds.id = f.document_id + where public.is_committed_artifact_generation(f.metadata, ds.metadata) + limit greatest($2 * 4, 48) + ), + trgm_matches as ( + select + f.id, + f.document_id, + f.source_chunk_id, + f.source_image_id, + f.page_number, + f.table_title, + f.row_label, + f.clinical_parameter, + f.threshold_value, + f.action, + (similarity(lower(coalesce(f.table_title, '') || ' ' || coalesce(f.row_label, '') || ' ' || coalesce(f.clinical_parameter, '')), q.normalized) * 0.8)::double precision as text_rank, + 'trgm_similarity'::text as match_reason + from query q + join public.document_table_facts f + on lower(coalesce(f.table_title, '') || ' ' || coalesce(f.row_label, '') || ' ' || coalesce(f.clinical_parameter, '')) % q.normalized + join doc_scope ds on ds.id = f.document_id + where public.is_committed_artifact_generation(f.metadata, ds.metadata) + and similarity(lower(coalesce(f.table_title, '') || ' ' || coalesce(f.row_label, '') || ' ' || coalesce(f.clinical_parameter, '')), q.normalized) >= 0.18 + order by similarity(lower(coalesce(f.table_title, '') || ' ' || coalesce(f.row_label, '') || ' ' || coalesce(f.clinical_parameter, '')), q.normalized) desc + limit greatest($2 * 4, 48) + ), + combined as ( + select * from fts_matches + union all + select * from term_matches + union all + select * from trgm_matches + ), + deduped as ( + select distinct on (c.id) + c.id, + c.document_id, + c.source_chunk_id, + c.source_image_id, + c.page_number, + c.table_title, + c.row_label, + c.clinical_parameter, + c.threshold_value, + c.action, + c.text_rank, + c.match_reason + from combined c + order by c.id, c.text_rank desc + ) + select + d.id, + d.document_id, + d.source_chunk_id, + d.source_image_id, + d.page_number, + d.table_title, + d.row_label, + d.clinical_parameter, + d.threshold_value, + d.action, + d.text_rank, + d.match_reason + from deduped d + where d.text_rank > 0 + order by d.text_rank desc, d.page_number asc nulls last + limit $2 + $body$ + using query_text, match_count, document_filters, owner_filter; +end; +$function$; + +revoke execute on function public.match_document_table_facts_text(text, integer, uuid[], uuid) + from public, anon, authenticated; +grant execute on function public.match_document_table_facts_text(text, integer, uuid[], uuid) to service_role; diff --git a/supabase/migrations/20260724130000_explicit_base_match_rpc_execute_grants.sql b/supabase/migrations/20260724130000_explicit_base_match_rpc_execute_grants.sql new file mode 100644 index 0000000000..3e9d320c76 --- /dev/null +++ b/supabase/migrations/20260724130000_explicit_base_match_rpc_execute_grants.sql @@ -0,0 +1,45 @@ +-- P3 hygiene from the 2026-07-24 Database↔Supabase interface audit: +-- base match RPCs historically relied on roles.sql / default-privilege churn for +-- EXECUTE lockdown. Reassert service_role-only grants next to the live signatures +-- so fresh replay and hosted apply stay explicit and idempotent. +-- +-- No retrieval behaviour change — grants only. + +do $$ +begin + if to_regprocedure('public.match_document_chunks(extensions.vector, integer, double precision, uuid, uuid)') is not null then + revoke execute on function public.match_document_chunks(extensions.vector, integer, double precision, uuid, uuid) + from public, anon, authenticated; + grant execute on function public.match_document_chunks(extensions.vector, integer, double precision, uuid, uuid) + to service_role; + end if; + + if to_regprocedure('public.match_document_chunks_hybrid(extensions.vector, text, integer, double precision, uuid[], uuid)') is not null then + revoke execute on function public.match_document_chunks_hybrid(extensions.vector, text, integer, double precision, uuid[], uuid) + from public, anon, authenticated; + grant execute on function public.match_document_chunks_hybrid(extensions.vector, text, integer, double precision, uuid[], uuid) + to service_role; + end if; + + if to_regprocedure('public.match_document_chunks_text(text, integer, uuid[], uuid)') is not null then + revoke execute on function public.match_document_chunks_text(text, integer, uuid[], uuid) + from public, anon, authenticated; + grant execute on function public.match_document_chunks_text(text, integer, uuid[], uuid) + to service_role; + end if; + + if to_regprocedure('public.match_document_memory_cards_hybrid(extensions.vector, text, integer, double precision, uuid[], uuid)') is not null then + revoke execute on function public.match_document_memory_cards_hybrid(extensions.vector, text, integer, double precision, uuid[], uuid) + from public, anon, authenticated; + grant execute on function public.match_document_memory_cards_hybrid(extensions.vector, text, integer, double precision, uuid[], uuid) + to service_role; + end if; + + if to_regprocedure('public.match_documents_for_query(text, integer, uuid)') is not null then + revoke execute on function public.match_documents_for_query(text, integer, uuid) + from public, anon, authenticated; + grant execute on function public.match_documents_for_query(text, integer, uuid) + to service_role; + end if; +end +$$; diff --git a/supabase/migrations/20260724130100_fix_invoke_ingestion_worker_url_to_guc.sql b/supabase/migrations/20260724130100_fix_invoke_ingestion_worker_url_to_guc.sql new file mode 100644 index 0000000000..a136fcd784 --- /dev/null +++ b/supabase/migrations/20260724130100_fix_invoke_ingestion_worker_url_to_guc.sql @@ -0,0 +1,63 @@ +-- P3 hygiene from the 2026-07-24 Database↔Supabase interface audit: +-- mirror invoke_indexing_v3_agent by reading the ingestion-worker base URL from +-- app.ingestion_worker_base_url, with the current project URL as fallback. +-- +-- Guarded: hosted Supabase denies ALTER DATABASE SET to the migration role +-- (42501). Swallow insufficient_privilege so the migration still succeeds; the +-- function falls back to the hardcoded URL via current_setting(..., true). + +do $$ +begin + execute format('alter database %I set app.ingestion_worker_base_url = %L', + current_database(), '[REDACTED]'); +exception + when insufficient_privilege then + raise notice 'Skipping ALTER DATABASE SET app.ingestion_worker_base_url (insufficient privilege on hosted Supabase); invoke_ingestion_worker falls back to the hardcoded URL.'; +end +$$; + +create or replace function public.invoke_ingestion_worker(p_limit integer default 25) +returns bigint +language plpgsql +security definer +set search_path = public, extensions, vault, pg_temp +as $$ +declare + v_request_id bigint; + v_jwt text; + v_limit integer := greatest(1, least(coalesce("p_limit", 25), 200)); + v_base_url text; +begin + select "decrypted_secret" into v_jwt + from "vault"."decrypted_secrets" + where "name" = 'cron_ingestion_jwt' + limit 1; + + if v_jwt is null or length(trim(v_jwt)) = 0 then + raise exception 'Missing Vault secret: cron_ingestion_jwt'; + end if; + + -- Prefer the GUC; fall back to the hardcoded production URL so that + -- existing deployments that have not yet set the GUC continue to work. + v_base_url := coalesce( + nullif(current_setting('app.ingestion_worker_base_url', true), ''), + '[REDACTED]' + ); + + select "net"."http_post"( + url := v_base_url || '/functions/v1/ingestion-worker?limit=' || v_limit::text, + headers := jsonb_build_object( + 'Content-Type','application/json', + 'Authorization','Bearer ' || v_jwt + ), + body := jsonb_build_object('source','pg_cron','worker','ingestion-worker','ts', now()), + timeout_milliseconds := 60000 + ) + into v_request_id; + + return v_request_id; +end; +$$; + +revoke execute on function public.invoke_ingestion_worker(integer) from public, anon, authenticated; +grant execute on function public.invoke_ingestion_worker(integer) to service_role; diff --git a/supabase/schema.sql b/supabase/schema.sql index e7f35428fc..6c3ee542d0 100644 --- a/supabase/schema.sql +++ b/supabase/schema.sql @@ -3470,108 +3470,19 @@ $function$; revoke execute on function public.purge_expired_rag_query_misses(integer) from public, anon, authenticated; grant execute on function public.purge_expired_rag_query_misses(integer) to service_role; -CREATE OR REPLACE FUNCTION public.correct_clinical_query_terms(input_query text, min_sim real DEFAULT 0.45) - RETURNS text - LANGUAGE plpgsql - STABLE SECURITY DEFINER - SET search_path TO 'public', 'extensions', 'pg_temp' - SET pg_trgm.similarity_threshold = 0.3 -AS $function$ -declare - tokens text[]; - tok text; - best text; - best_sim real; - vocab text[]; - corrected text[] := array[]::text[]; - changed boolean := false; +-- Prefer app.ingestion_worker_base_url (GUC) with the hardcoded project URL as +-- fallback, matching invoke_indexing_v3_agent. Guarded ALTER DATABASE SET so +-- hosted Supabase (42501) still replays cleanly. +do $$ begin - if min_sim is null or min_sim < 0.3 or min_sim > 1 then - raise exception 'min_sim must be between 0.3 and 1.0' using errcode = '22023'; - end if; - - if input_query is null or length(trim(input_query)) = 0 then - return input_query; - end if; - - -- Build the known-term vocabulary once per call. Every source is scoped to the - -- public (null-owner) corpus: this function is SECURITY DEFINER and bypasses RLS, and - -- both rag_aliases and documents carry owner-scoped private rows (deep-memory persists - -- owner-scoped aliases/canonicals), so an unscoped read would leak private-document - -- terms across tenants. Mirrors migration 20260717120000_corrector_public_titles_only. - select array_agg(distinct term) into vocab - from ( - select lower(alias) as term from public.rag_aliases where enabled and owner_id is null and length(alias) between 4 and 40 - union - select lower(canonical) from public.rag_aliases where enabled and owner_id is null and length(canonical) between 4 and 40 - union - select w from public.documents d, lateral unnest(regexp_split_to_array(lower(d.title), '[^a-z]+')) as w - where d.status = 'indexed' and d.owner_id is null and length(w) between 4 and 40 - ) t; - - tokens := regexp_split_to_array(lower(input_query), '\s+'); - foreach tok in array tokens loop - if length(tok) < 4 then - corrected := corrected || tok; - continue; - end if; - best := null; - best_sim := 0; - select candidate.term, similarity(candidate.term, tok) - into best, best_sim - from ( - ( - select lower(alias) as term - from public.rag_aliases - where enabled - and length(alias) between 4 and 40 - and lower(alias) % tok - order by similarity(lower(alias), tok) desc, lower(alias) - limit 32 - ) - union all - ( - select lower(canonical) as term - from public.rag_aliases - where enabled - and length(canonical) between 4 and 40 - and lower(canonical) % tok - order by similarity(lower(canonical), tok) desc, lower(canonical) - limit 32 - ) - union all - ( - select word as term - from public.document_title_words - where length(word) between 4 and 40 - and word % tok - order by similarity(word, tok) desc, word - limit 32 - ) - ) candidate - order by similarity(candidate.term, tok) desc, candidate.term - limit 1; - if best is not null and best_sim >= min_sim and best <> tok and length(best) >= length(tok) then - corrected := corrected || best; - changed := true; - else - corrected := corrected || tok; - end if; - end loop; - - if not changed then - return input_query; - end if; - return array_to_string(corrected, ' '); -end; -$function$; - -revoke execute on function public.correct_clinical_query_terms(text, real) from public, anon, authenticated; -grant execute on function public.correct_clinical_query_terms(text, real) to service_role; + execute format('alter database %I set app.ingestion_worker_base_url = %L', + current_database(), '[REDACTED]'); +exception + when insufficient_privilege then + raise notice 'Skipping ALTER DATABASE SET app.ingestion_worker_base_url (insufficient privilege on hosted Supabase).'; +end +$$; --- NOTE: unlike invoke_indexing_v3_agent (URL moved to a GUC by 20260702160000), --- the live definition still hardcodes the project URL. Codified as-is; migrate --- to the GUC pattern in a follow-up if this RPC stays. CREATE OR REPLACE FUNCTION public.invoke_ingestion_worker(p_limit integer DEFAULT 25) RETURNS bigint LANGUAGE plpgsql @@ -3582,6 +3493,7 @@ declare v_request_id bigint; v_jwt text; v_limit integer := greatest(1, least(coalesce("p_limit", 25), 200)); + v_base_url text; begin select "decrypted_secret" into v_jwt from "vault"."decrypted_secrets" @@ -3592,8 +3504,13 @@ begin raise exception 'Missing Vault secret: cron_ingestion_jwt'; end if; + v_base_url := coalesce( + nullif(current_setting('app.ingestion_worker_base_url', true), ''), + '[REDACTED]' + ); + select "net"."http_post"( - url := 'https://sjrfecxgysukkwxsowpy.supabase.co/functions/v1/ingestion-worker?limit=' || v_limit::text, + url := v_base_url || '/functions/v1/ingestion-worker?limit=' || v_limit::text, headers := jsonb_build_object( 'Content-Type','application/json', 'Authorization','Bearer ' || v_jwt @@ -6506,23 +6423,25 @@ drop function if exists public.match_document_table_facts_text(text, integer, uu CREATE OR REPLACE FUNCTION public.match_document_table_facts_text(query_text text, match_count integer DEFAULT 16, document_filters uuid[] DEFAULT NULL::uuid[], owner_filter uuid DEFAULT NULL::uuid) RETURNS TABLE(id uuid, document_id uuid, source_chunk_id uuid, source_image_id uuid, page_number integer, table_title text, row_label text, clinical_parameter text, threshold_value text, action text, text_rank double precision, match_reason text) - LANGUAGE sql + LANGUAGE plpgsql STABLE SET search_path TO 'public', 'extensions', 'pg_temp' SET plan_cache_mode TO 'force_custom_plan' AS $function$ +begin + return query execute $body$ with query as ( select - websearch_to_tsquery('english', coalesce(query_text, '')) as tsq, - lower(trim(regexp_replace(coalesce(query_text, ''), '\\s+', ' ', 'g'))) as normalized, - string_to_array(lower(trim(regexp_replace(coalesce(query_text, ''), '\\s+', ' ', 'g'))), ' ')::text[] as tokens + websearch_to_tsquery('english', coalesce($1, '')) as tsq, + lower(trim(regexp_replace(coalesce($1, ''), '\s+', ' ', 'g'))) as normalized, + string_to_array(lower(trim(regexp_replace(coalesce($1, ''), '\s+', ' ', 'g'))), ' ')::text[] as tokens ), doc_scope as ( select d.id, d.metadata from public.documents d where d.status = 'indexed' - and public.retrieval_owner_matches(owner_filter, d.owner_id) - and (document_filters is null or d.id = any(document_filters)) + and public.retrieval_owner_matches($4, d.owner_id) + and ($3 is null or d.id = any($3)) ), fts_matches as ( select @@ -6547,7 +6466,7 @@ AS $function$ join doc_scope ds on ds.id = f.document_id where public.is_committed_artifact_generation(f.metadata, ds.metadata) order by ts_rank_cd(f.search_tsv, q.tsq) desc - limit greatest(match_count * 5, 64) + limit greatest($2 * 5, 64) ), term_matches as ( select @@ -6569,7 +6488,7 @@ AS $function$ and f.normalized_terms && q.tokens join doc_scope ds on ds.id = f.document_id where public.is_committed_artifact_generation(f.metadata, ds.metadata) - limit greatest(match_count * 4, 48) + limit greatest($2 * 4, 48) ), trgm_matches as ( select @@ -6592,7 +6511,7 @@ AS $function$ where public.is_committed_artifact_generation(f.metadata, ds.metadata) and similarity(lower(coalesce(f.table_title, '') || ' ' || coalesce(f.row_label, '') || ' ' || coalesce(f.clinical_parameter, '')), q.normalized) >= 0.18 order by similarity(lower(coalesce(f.table_title, '') || ' ' || coalesce(f.row_label, '') || ' ' || coalesce(f.clinical_parameter, '')), q.normalized) desc - limit greatest(match_count * 4, 48) + limit greatest($2 * 4, 48) ), combined as ( select * from fts_matches @@ -6634,7 +6553,10 @@ AS $function$ from deduped d where d.text_rank > 0 order by d.text_rank desc, d.page_number asc nulls last - limit match_count; + limit $2 + $body$ + using query_text, match_count, document_filters, owner_filter; +end; $function$; CREATE OR REPLACE FUNCTION public.match_documents_for_query(query_text text, match_count integer DEFAULT 12, owner_filter uuid DEFAULT NULL::uuid) @@ -7261,6 +7183,33 @@ revoke execute on function public.match_document_table_facts_text(text, integer, from public, anon, authenticated; grant execute on function public.match_document_table_facts_text(text, integer, uuid[], uuid) to service_role; +-- Base match RPCs: keep EXECUTE lockdown explicit in the canonical snapshot so +-- fresh replay does not depend solely on roles.sql / default-privilege churn. +revoke execute on function public.match_document_chunks(extensions.vector, integer, double precision, uuid, uuid) + from public, anon, authenticated; +grant execute on function public.match_document_chunks(extensions.vector, integer, double precision, uuid, uuid) + to service_role; + +revoke execute on function public.match_document_chunks_hybrid(extensions.vector, text, integer, double precision, uuid[], uuid) + from public, anon, authenticated; +grant execute on function public.match_document_chunks_hybrid(extensions.vector, text, integer, double precision, uuid[], uuid) + to service_role; + +revoke execute on function public.match_document_chunks_text(text, integer, uuid[], uuid) + from public, anon, authenticated; +grant execute on function public.match_document_chunks_text(text, integer, uuid[], uuid) + to service_role; + +revoke execute on function public.match_document_memory_cards_hybrid(extensions.vector, text, integer, double precision, uuid[], uuid) + from public, anon, authenticated; +grant execute on function public.match_document_memory_cards_hybrid(extensions.vector, text, integer, double precision, uuid[], uuid) + to service_role; + +revoke execute on function public.match_documents_for_query(text, integer, uuid) + from public, anon, authenticated; +grant execute on function public.match_documents_for_query(text, integer, uuid) + to service_role; + revoke execute on function public.repair_enrichment_quality_batch(integer) from public, anon, authenticated; grant execute on function public.repair_enrichment_quality_batch(integer) to service_role; diff --git a/tests/reindex-enrichment-lease.test.ts b/tests/reindex-enrichment-lease.test.ts new file mode 100644 index 0000000000..523269b830 --- /dev/null +++ b/tests/reindex-enrichment-lease.test.ts @@ -0,0 +1,333 @@ +import { afterEach, describe, expect, it, vi } from "vitest"; + +const documentId = "11111111-1111-4111-8111-111111111111"; +const otherDocumentId = "33333333-3333-4333-8333-333333333333"; +const userId = "22222222-2222-4222-8222-222222222222"; + +afterEach(() => { + vi.resetModules(); + vi.restoreAllMocks(); +}); + +function readySafety() { + return { + ok: true as const, + checkedAt: "2026-07-24T00:00:00.000Z", + reason: "ready" as const, + message: "safe", + activeJobs: [] as [], + staleProcessingJobs: [] as [], + }; +} + +function documentRow(id = documentId) { + return { + id, + owner_id: userId, + title: "Lease Guard Protocol", + file_name: "lease-guard.pdf", + source_path: null, + import_batch_id: null, + status: "indexed", + error_message: null, + page_count: 2, + chunk_count: 4, + image_count: 0, + metadata: {}, + }; +} + +function mockAuthenticatedAdmin() { + vi.doMock("@/lib/supabase/auth", () => ({ + AuthenticationError: class AuthenticationError extends Error {}, + requireAuthenticatedUser: vi.fn(async () => ({ id: userId })), + unauthorizedResponse: vi.fn(), + })); +} + +describe("reindex enrichment-lease gate (#052)", () => { + it("blocks full single-document reindex while a fresh agent enrichment lease is live", async () => { + const rpc = vi.fn(); + const from = vi.fn(() => ({ + select: vi.fn().mockReturnThis(), + eq: vi.fn().mockReturnThis(), + maybeSingle: vi.fn(async () => ({ data: documentRow(), error: null })), + update: vi.fn().mockReturnThis(), + insert: vi.fn().mockReturnThis(), + })); + const checkIngestionMutationSafety = vi.fn(async () => readySafety()); + const hasActiveAgentEnrichmentJob = vi.fn(async () => true); + + vi.doMock("@/lib/env", () => ({ + env: { WORKER_STALE_AFTER_MINUTES: 15, WORKER_MAX_ATTEMPTS: 3 }, + isDemoMode: () => false, + })); + vi.doMock("@/lib/supabase/admin", () => ({ createAdminClient: () => ({ from, rpc }) })); + mockAuthenticatedAdmin(); + vi.doMock("@/lib/api-rate-limit", () => ({ + consumeApiRateLimit: vi.fn(async () => ({ limited: false })), + rateLimitJsonResponse: vi.fn(), + })); + vi.doMock("@/lib/ingestion-mutation-safety", () => ({ + checkIngestionMutationSafety, + hasActiveAgentEnrichmentJob, + ingestionMutationSafetyPayload: vi.fn((safety) => ({ error: safety.message, safety })), + activeIngestionJobColumns: "id", + buildActiveJobsSafetyResult: vi.fn(), + ingestionRollbackFenceStamp: vi.fn(() => "2026-07-24T00:00:00.000000Z"), + })); + + const { POST } = await import("../src/app/api/documents/[id]/reindex/route"); + const response = await POST( + new Request(`http://localhost/api/documents/${documentId}/reindex`, { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ mode: "full" }), + }), + { params: Promise.resolve({ id: documentId }) }, + ); + + expect(response.status).toBe(409); + expect(await response.json()).toMatchObject({ error: "Reindex is paused while enrichment is active." }); + expect(hasActiveAgentEnrichmentJob).toHaveBeenCalledWith(expect.objectContaining({ documentId })); + expect(rpc).not.toHaveBeenCalled(); + }); + + it("does not consult the enrichment lease gate for enrichment-mode reindex", async () => { + const rpc = vi.fn(async () => ({ data: { job_id: "enrichment-job" }, error: null })); + const from = vi.fn(() => ({ + select: vi.fn().mockReturnThis(), + eq: vi.fn().mockReturnThis(), + maybeSingle: vi.fn(async () => ({ data: documentRow(), error: null })), + })); + const checkIngestionMutationSafety = vi.fn(async () => readySafety()); + const hasActiveAgentEnrichmentJob = vi.fn(async () => true); + + vi.doMock("@/lib/env", () => ({ + env: { WORKER_STALE_AFTER_MINUTES: 15, WORKER_MAX_ATTEMPTS: 3 }, + isDemoMode: () => false, + })); + vi.doMock("@/lib/supabase/admin", () => ({ createAdminClient: () => ({ from, rpc }) })); + mockAuthenticatedAdmin(); + vi.doMock("@/lib/api-rate-limit", () => ({ + consumeApiRateLimit: vi.fn(async () => ({ limited: false })), + rateLimitJsonResponse: vi.fn(), + })); + vi.doMock("@/lib/ingestion-mutation-safety", () => ({ + checkIngestionMutationSafety, + hasActiveAgentEnrichmentJob, + ingestionMutationSafetyPayload: vi.fn((safety) => ({ error: safety.message, safety })), + activeIngestionJobColumns: "id", + buildActiveJobsSafetyResult: vi.fn(), + ingestionRollbackFenceStamp: vi.fn(() => "2026-07-24T00:00:00.000000Z"), + })); + + const { POST } = await import("../src/app/api/documents/[id]/reindex/route"); + const response = await POST( + new Request(`http://localhost/api/documents/${documentId}/reindex`, { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ mode: "enrichment" }), + }), + { params: Promise.resolve({ id: documentId }) }, + ); + + expect(response.status).toBe(202); + expect(hasActiveAgentEnrichmentJob).not.toHaveBeenCalled(); + expect(rpc).toHaveBeenCalledWith( + "request_indexing_v3_enrichment", + expect.objectContaining({ p_document_id: documentId, p_owner_id: userId }), + ); + }); + + it("blocks bulk full reindex when any selected document has a fresh enrichment lease", async () => { + const rpc = vi.fn(); + const documentsQuery = { + select: vi.fn(), + eq: vi.fn(), + in: vi.fn(async () => ({ + data: [documentRow(documentId), documentRow(otherDocumentId)], + error: null, + })), + }; + documentsQuery.select.mockReturnValue(documentsQuery); + documentsQuery.eq.mockReturnValue(documentsQuery); + const from = vi.fn(() => documentsQuery); + + const checkIngestionMutationSafety = vi.fn(async () => readySafety()); + const listDocumentsWithActiveAgentEnrichment = vi.fn(async () => [documentId]); + + vi.doMock("@/lib/env", () => ({ + env: { WORKER_STALE_AFTER_MINUTES: 15, WORKER_MAX_ATTEMPTS: 3 }, + isDemoMode: () => false, + })); + vi.doMock("@/lib/supabase/admin", () => ({ createAdminClient: () => ({ from, rpc }) })); + mockAuthenticatedAdmin(); + vi.doMock("@/lib/api-rate-limit", () => ({ + consumeApiRateLimit: vi.fn(async () => ({ limited: false })), + rateLimitJsonResponse: vi.fn(), + })); + vi.doMock("@/lib/ingestion-mutation-safety", () => ({ + checkIngestionMutationSafety, + listDocumentsWithActiveAgentEnrichment, + ingestionMutationSafetyPayload: vi.fn((safety) => ({ error: safety.message, safety })), + activeIngestionJobColumns: "id", + buildActiveJobsSafetyResult: vi.fn(), + ingestionRollbackFenceStamp: vi.fn(() => "2026-07-24T00:00:00.000000Z"), + })); + vi.doMock("@/lib/rag/rag", () => ({ invalidateRagCachesForOwner: vi.fn() })); + + const { POST } = await import("../src/app/api/documents/bulk/reindex/route"); + const response = await POST( + new Request("http://localhost/api/documents/bulk/reindex", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ documentIds: [documentId, otherDocumentId], mode: "full" }), + }), + ); + + expect(response.status).toBe(409); + expect(await response.json()).toMatchObject({ + error: "Bulk reindex is paused while enrichment is active for one or more selected documents.", + blockedDocumentIds: [documentId], + }); + expect(listDocumentsWithActiveAgentEnrichment).toHaveBeenCalledWith( + expect.objectContaining({ documentIds: expect.arrayContaining([documentId, otherDocumentId]) }), + ); + expect(rpc).not.toHaveBeenCalled(); + }); + + it("keeps bulk enrichment mode free of the full-reindex lease preflight", async () => { + const rpc = vi.fn(async () => ({ data: { job_id: "enrichment-job" }, error: null })); + const documentsQuery = { + select: vi.fn(), + eq: vi.fn(), + in: vi.fn(async () => ({ data: [documentRow()], error: null })), + }; + documentsQuery.select.mockReturnValue(documentsQuery); + documentsQuery.eq.mockReturnValue(documentsQuery); + const from = vi.fn(() => documentsQuery); + const checkIngestionMutationSafety = vi.fn(async () => readySafety()); + const listDocumentsWithActiveAgentEnrichment = vi.fn(async () => [documentId]); + + vi.doMock("@/lib/env", () => ({ + env: { WORKER_STALE_AFTER_MINUTES: 15, WORKER_MAX_ATTEMPTS: 3 }, + isDemoMode: () => false, + })); + vi.doMock("@/lib/supabase/admin", () => ({ createAdminClient: () => ({ from, rpc }) })); + mockAuthenticatedAdmin(); + vi.doMock("@/lib/api-rate-limit", () => ({ + consumeApiRateLimit: vi.fn(async () => ({ limited: false })), + rateLimitJsonResponse: vi.fn(), + })); + vi.doMock("@/lib/ingestion-mutation-safety", () => ({ + checkIngestionMutationSafety, + listDocumentsWithActiveAgentEnrichment, + ingestionMutationSafetyPayload: vi.fn((safety) => ({ error: safety.message, safety })), + activeIngestionJobColumns: "id", + buildActiveJobsSafetyResult: vi.fn(), + ingestionRollbackFenceStamp: vi.fn(() => "2026-07-24T00:00:00.000000Z"), + })); + vi.doMock("@/lib/rag/rag", () => ({ invalidateRagCachesForOwner: vi.fn() })); + + const { POST } = await import("../src/app/api/documents/bulk/reindex/route"); + const response = await POST( + new Request("http://localhost/api/documents/bulk/reindex", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ documentIds: [documentId], mode: "enrichment" }), + }), + ); + + expect(response.status).toBe(200); + expect(listDocumentsWithActiveAgentEnrichment).not.toHaveBeenCalled(); + expect(rpc).toHaveBeenCalledWith( + "request_indexing_v3_enrichment", + expect.objectContaining({ p_document_id: documentId }), + ); + }); + + it("scopes retry_failed enrichment lease checks to failed documents only", async () => { + const failedId = documentId; + const indexedLeasedId = otherDocumentId; + const rpc = vi.fn(); + const documentsQuery = { + select: vi.fn(), + eq: vi.fn(), + in: vi.fn(async () => ({ + data: [ + { ...documentRow(failedId), status: "failed" }, + { ...documentRow(indexedLeasedId), status: "indexed" }, + ], + error: null, + })), + }; + documentsQuery.select.mockReturnValue(documentsQuery); + documentsQuery.eq.mockReturnValue(documentsQuery); + + const from = vi.fn((table: string) => { + if (table === "documents") { + return { + ...documentsQuery, + update: vi.fn(() => ({ + eq: vi.fn().mockReturnThis(), + select: vi.fn().mockReturnThis(), + maybeSingle: vi.fn(async () => ({ data: { id: failedId }, error: null })), + })), + }; + } + return { + select: vi.fn().mockReturnThis(), + eq: vi.fn().mockReturnThis(), + in: vi.fn().mockReturnThis(), + order: vi.fn().mockReturnThis(), + limit: vi.fn(async () => ({ data: [], error: null })), + insert: vi.fn(async () => ({ data: [{ id: "job-1" }], error: null })), + }; + }); + + const checkIngestionMutationSafety = vi.fn(async () => readySafety()); + const listDocumentsWithActiveAgentEnrichment = vi.fn(async () => []); + + vi.doMock("@/lib/env", () => ({ + env: { WORKER_STALE_AFTER_MINUTES: 15, WORKER_MAX_ATTEMPTS: 3 }, + isDemoMode: () => false, + })); + vi.doMock("@/lib/supabase/admin", () => ({ createAdminClient: () => ({ from, rpc }) })); + mockAuthenticatedAdmin(); + vi.doMock("@/lib/api-rate-limit", () => ({ + consumeApiRateLimit: vi.fn(async () => ({ limited: false })), + rateLimitJsonResponse: vi.fn(), + })); + vi.doMock("@/lib/ingestion-mutation-safety", () => ({ + checkIngestionMutationSafety, + listDocumentsWithActiveAgentEnrichment, + ingestionMutationSafetyPayload: vi.fn((safety) => ({ error: safety.message, safety })), + activeIngestionJobColumns: "id", + buildActiveJobsSafetyResult: vi.fn(), + ingestionRollbackFenceStamp: vi.fn(() => "2026-07-24T00:00:00.000000Z"), + })); + vi.doMock("@/lib/rag/rag", () => ({ invalidateRagCachesForOwner: vi.fn() })); + vi.doMock("@/lib/reindex-pipeline", () => ({ isAtomicReindexCandidate: () => true })); + + const { POST } = await import("../src/app/api/documents/bulk/reindex/route"); + const response = await POST( + new Request("http://localhost/api/documents/bulk/reindex", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ documentIds: [failedId, indexedLeasedId], mode: "retry_failed" }), + }), + ); + + expect(response.status).toBe(200); + expect(listDocumentsWithActiveAgentEnrichment).toHaveBeenCalledWith( + expect.objectContaining({ documentIds: [failedId] }), + ); + const body = await response.json(); + expect(body.results).toEqual( + expect.arrayContaining([ + expect.objectContaining({ documentId: indexedLeasedId, ok: false, error: "Document is not failed." }), + ]), + ); + }); +}); diff --git a/tests/stranded-queued-recovery.test.ts b/tests/stranded-queued-recovery.test.ts new file mode 100644 index 0000000000..a1d39853de --- /dev/null +++ b/tests/stranded-queued-recovery.test.ts @@ -0,0 +1,235 @@ +import { beforeEach, describe, expect, it, vi } from "vitest"; + +const enqueueDocumentReindexJob = vi.hoisted(() => vi.fn()); + +vi.mock("@/lib/ingestion-enqueue", () => ({ + enqueueDocumentReindexJob, +})); + +import { + isStrandedQueuedDocument, + listStrandedQueuedDocuments, + recoverStrandedQueuedDocuments, +} from "../src/lib/stranded-queued-recovery"; + +beforeEach(() => { + enqueueDocumentReindexJob.mockReset(); +}); + +describe("stranded queued-without-job recovery (#062)", () => { + const now = new Date("2026-07-24T12:00:00.000Z"); + + it("reproduces the upload crash window: queued document with zero open jobs is stranded once aged", () => { + // Upload inserts documents then ingestion_jobs as two PostgREST calls + // (src/app/api/upload/route.ts). A process kill between them leaves this shape. + const afterDocumentInsert = { + status: "queued" as const, + created_at: "2026-07-24T11:30:00.000Z", + updated_at: "2026-07-24T11:30:00.000Z", + }; + expect( + isStrandedQueuedDocument({ + document: afterDocumentInsert, + openJobCount: 0, + minAgeMinutes: 15, + now, + }), + ).toBe(true); + }); + + it("does not treat a young queued row as stranded (in-flight upload still finishing)", () => { + expect( + isStrandedQueuedDocument({ + document: { + status: "queued", + created_at: "2026-07-24T11:55:00.000Z", + updated_at: "2026-07-24T11:55:00.000Z", + }, + openJobCount: 0, + minAgeMinutes: 15, + now, + }), + ).toBe(false); + }); + + it("does not treat a queued row that already has an open job as stranded", () => { + expect( + isStrandedQueuedDocument({ + document: { + status: "queued", + created_at: "2026-07-24T11:00:00.000Z", + updated_at: "2026-07-24T11:00:00.000Z", + }, + openJobCount: 1, + minAgeMinutes: 15, + now, + }), + ).toBe(false); + }); + + it("lists only aged queued documents that lack open jobs, scoped by owner when provided", async () => { + const ownerId = "22222222-2222-4222-8222-222222222222"; + const stranded = { + id: "11111111-1111-4111-8111-111111111111", + owner_id: ownerId, + status: "queued", + error_message: null, + page_count: 0, + chunk_count: 0, + image_count: 0, + import_batch_id: null, + created_at: "2026-07-24T11:00:00.000Z", + updated_at: "2026-07-24T11:00:00.000Z", + }; + const withOpenJob = { + ...stranded, + id: "33333333-3333-4333-8333-333333333333", + }; + + const filters: Array<{ column: string; value: unknown }> = []; + const documentsQuery: Record = {}; + documentsQuery.select = vi.fn(() => documentsQuery); + documentsQuery.eq = vi.fn((column: string, value: unknown) => { + filters.push({ column, value }); + return documentsQuery; + }); + documentsQuery.lt = vi.fn(() => documentsQuery); + documentsQuery.order = vi.fn(() => documentsQuery); + documentsQuery.range = vi.fn(async () => ({ data: [stranded, withOpenJob], error: null })); + + let jobInCalls = 0; + const jobsQuery: Record = {}; + jobsQuery.select = vi.fn(() => jobsQuery); + jobsQuery.in = vi.fn(() => { + jobInCalls += 1; + if (jobInCalls >= 2) { + return Promise.resolve({ data: [{ document_id: withOpenJob.id }], error: null }); + } + return jobsQuery; + }); + + const supabase = { + from: vi.fn((table: string) => (table === "documents" ? documentsQuery : jobsQuery)), + }; + + const listed = await listStrandedQueuedDocuments({ + supabase: supabase as never, + ownerId, + minAgeMinutes: 15, + limit: 20, + now, + }); + + expect(filters).toContainEqual({ column: "owner_id", value: ownerId }); + expect(filters).toContainEqual({ column: "status", value: "queued" }); + expect(listed.map((document) => document.id)).toEqual([stranded.id]); + }); + + it("pages past open-job queued rows so stranded candidates are not starved by the limit", async () => { + const ownerId = "22222222-2222-4222-8222-222222222222"; + const stranded = { + id: "11111111-1111-4111-8111-111111111111", + owner_id: ownerId, + status: "queued", + error_message: null, + page_count: 0, + chunk_count: 0, + image_count: 0, + import_batch_id: null, + created_at: "2026-07-24T11:00:00.000Z", + updated_at: "2026-07-24T11:00:00.000Z", + }; + const openJobIds = Array.from( + { length: 20 }, + (_, index) => `33333333-3333-4333-8333-${String(index).padStart(12, "0")}`, + ); + const openJobRows = openJobIds.map((id, index) => ({ + ...stranded, + id, + created_at: `2026-07-24T10:${String(index).padStart(2, "0")}:00.000Z`, + updated_at: `2026-07-24T10:${String(index).padStart(2, "0")}:00.000Z`, + })); + + let page = 0; + const documentsQuery: Record = {}; + documentsQuery.select = vi.fn(() => documentsQuery); + documentsQuery.eq = vi.fn(() => documentsQuery); + documentsQuery.lt = vi.fn(() => documentsQuery); + documentsQuery.order = vi.fn(() => documentsQuery); + documentsQuery.range = vi.fn(async () => { + page += 1; + if (page === 1) return { data: openJobRows, error: null }; + if (page === 2) return { data: [stranded], error: null }; + return { data: [], error: null }; + }); + + let pendingDocumentIds: string[] = []; + const jobsQuery: Record = {}; + jobsQuery.select = vi.fn(() => jobsQuery); + jobsQuery.in = vi.fn((column: string, values: string[]) => { + if (column === "document_id") { + pendingDocumentIds = [...values]; + return jobsQuery; + } + return Promise.resolve({ + data: pendingDocumentIds.filter((id) => openJobIds.includes(id)).map((document_id) => ({ document_id })), + error: null, + }); + }); + + const supabase = { + from: vi.fn((table: string) => (table === "documents" ? documentsQuery : jobsQuery)), + }; + + const listed = await listStrandedQueuedDocuments({ + supabase: supabase as never, + ownerId, + minAgeMinutes: 15, + limit: 1, + now, + }); + + expect(page).toBeGreaterThanOrEqual(2); + expect(listed.map((document) => document.id)).toEqual([stranded.id]); + }); + + it("recovers a stranded row by enqueueing exactly one job and treats already-active as idempotent", async () => { + enqueueDocumentReindexJob + .mockResolvedValueOnce({ outcome: "enqueued", job: { id: "job-1" } }) + .mockResolvedValueOnce({ outcome: "already_active" }); + + const docs = [ + { + id: "11111111-1111-4111-8111-111111111111", + owner_id: "22222222-2222-4222-8222-222222222222", + status: "queued", + error_message: null, + page_count: 0, + chunk_count: 0, + image_count: 0, + import_batch_id: null, + created_at: "2026-07-24T11:00:00.000Z", + updated_at: "2026-07-24T11:00:00.000Z", + }, + { + id: "33333333-3333-4333-8333-333333333333", + owner_id: "22222222-2222-4222-8222-222222222222", + status: "queued", + error_message: null, + page_count: 0, + chunk_count: 0, + image_count: 0, + import_batch_id: null, + created_at: "2026-07-24T11:00:00.000Z", + updated_at: "2026-07-24T11:00:00.000Z", + }, + ]; + + const results = await recoverStrandedQueuedDocuments({ supabase: {} as never, documents: docs }); + expect(results).toEqual([ + { documentId: docs[0].id, outcome: "enqueued", jobId: "job-1" }, + { documentId: docs[1].id, outcome: "already_active" }, + ]); + expect(enqueueDocumentReindexJob).toHaveBeenCalledTimes(2); + }); +}); diff --git a/tests/supabase-schema.test.ts b/tests/supabase-schema.test.ts index e4847469a5..cfd6d03d1c 100644 --- a/tests/supabase-schema.test.ts +++ b/tests/supabase-schema.test.ts @@ -235,6 +235,14 @@ const publicTitleCorrectorMigration = readFileSync( new URL("../supabase/migrations/20260717171000_public_title_corrector.sql", import.meta.url), "utf8", ).replace(/\s+/g, " "); +const baseMatchRpcExecuteGrantsMigration = readFileSync( + new URL("../supabase/migrations/20260724130000_explicit_base_match_rpc_execute_grants.sql", import.meta.url), + "utf8", +).replace(/\s+/g, " "); +const invokeIngestionWorkerGucMigration = readFileSync( + new URL("../supabase/migrations/20260724130100_fix_invoke_ingestion_worker_url_to_guc.sql", import.meta.url), + "utf8", +).replace(/\s+/g, " "); function finalSqlSegment(sql: string, startMarker: string, endMarker: string) { const normalized = sql.toLowerCase(); @@ -1649,6 +1657,26 @@ describe("Supabase Preview replay guards", () => { ); expect(tableFactsRpc).toContain(`${rpcExpression} % q.normalized`); }); + + it("plans table-facts text matching via plpgsql EXECUTE for per-call custom plans", () => { + const tableFactsMigration = readFileSync( + new URL("../supabase/migrations/20260724120000_table_facts_plpgsql_execute.sql", import.meta.url), + "utf8", + ); + const tableFactsRpc = finalSqlSegment( + schema, + "create or replace function public.match_document_table_facts_text(", + "$function$;", + ); + expect(tableFactsRpc).toContain("language plpgsql"); + expect(tableFactsRpc).toContain("return query execute"); + expect(tableFactsRpc).toContain("using query_text, match_count, document_filters, owner_filter"); + expect(tableFactsMigration).toContain("language plpgsql"); + expect(tableFactsMigration).toContain("return query execute"); + expect(tableFactsMigration).toContain( + "revoke execute on function public.match_document_table_facts_text(text, integer, uuid[], uuid)", + ); + }); }); describe("Clinical query-term corrector — tenant-safe vocabulary (F10)", () => { @@ -1701,4 +1729,64 @@ describe("Clinical query-term corrector — tenant-safe vocabulary (F10)", () => "grant execute on function public.correct_clinical_query_terms(text, real) to service_role;", ); }); + + it("defines correct_clinical_query_terms exactly once with owner-scoped rag_aliases probes", () => { + // Schema hygiene: a superseded unscoped duplicate previously lived earlier in + // schema.sql and only lost because CREATE OR REPLACE order favored the later + // definition. Pin a single authoritative body so reorder/extract cannot revive + // the cross-tenant alias side-channel. + const definitionMatches = schema.match(/create or replace function public\.correct_clinical_query_terms/gi); + expect(definitionMatches).toHaveLength(1); + + const corrector = finalSqlSegment( + schema, + "create or replace function public.correct_clinical_query_terms", + "revoke execute on function public.correct_clinical_query_terms", + ); + const aliasProbeBlocks = corrector.match(/from public\.rag_aliases[\s\S]*?limit 32/gi) ?? []; + expect(aliasProbeBlocks.length).toBeGreaterThanOrEqual(2); + for (const block of aliasProbeBlocks) { + expect(block).toMatch(/owner_id\s+is\s+null/i); + } + expect(corrector).not.toContain("array_agg(distinct term)"); + expect(corrector).not.toContain("unnest(vocab)"); + }); + + it("keeps base match RPC execute grants explicit in schema and the forward migration", () => { + // 2026-07-24 interface audit P3: do not rely solely on roles.sql / default + // privilege churn for the live match_* signatures. + for (const sql of [schema, baseMatchRpcExecuteGrantsMigration]) { + expect(sql).toContain( + "revoke execute on function public.match_document_chunks(extensions.vector, integer, double precision, uuid, uuid)", + ); + expect(sql).toContain( + "grant execute on function public.match_document_chunks(extensions.vector, integer, double precision, uuid, uuid)", + ); + expect(sql).toContain( + "revoke execute on function public.match_document_chunks_hybrid(extensions.vector, text, integer, double precision, uuid[], uuid)", + ); + expect(sql).toContain( + "revoke execute on function public.match_document_chunks_text(text, integer, uuid[], uuid)", + ); + expect(sql).toContain( + "revoke execute on function public.match_document_memory_cards_hybrid(extensions.vector, text, integer, double precision, uuid[], uuid)", + ); + expect(sql).toContain("revoke execute on function public.match_documents_for_query(text, integer, uuid)"); + expect(sql).toContain("grant execute on function public.match_documents_for_query(text, integer, uuid)"); + } + }); + + it("moves invoke_ingestion_worker to the GUC base-URL pattern with service-role-only execute", () => { + for (const sql of [schema, invokeIngestionWorkerGucMigration]) { + expect(sql).toContain("alter database %I set app.ingestion_worker_base_url = %L"); + expect(sql).toContain("when insufficient_privilege then"); + expect(sql).toContain("nullif(current_setting('app.ingestion_worker_base_url', true), '')"); + expect(sql).toContain("v_base_url || '/functions/v1/ingestion-worker?limit='"); + expect(sql).not.toContain("[REDACTED]/functions/v1/ingestion-worker"); + expect(sql).toContain( + "revoke execute on function public.invoke_ingestion_worker(integer) from public, anon, authenticated", + ); + expect(sql).toContain("grant execute on function public.invoke_ingestion_worker(integer) to service_role"); + } + }); });