diff --git a/docs/branch-review-ledger.md b/docs/branch-review-ledger.md index 60b76fd658..f65383b769 100644 --- a/docs/branch-review-ledger.md +++ b/docs/branch-review-ledger.md @@ -392,3 +392,9 @@ Use this ledger to prevent repeated branch and PR reviews when the reviewed HEAD | 2026-07-13 | origin/codex/rag-canary-completion | d1b7647bde73e7d0558472ad6aa19c3b49d94639 | branch-cleanup | Retained for open PR #612 and active worktree activity. | Fresh GitHub open-PR query and moving-head observation; deletion prohibited. | | 2026-07-13 | origin/main | 528a1752f41cd29a518ca9341c93c724030173ae | branch-cleanup | Protected base branch retained. | Final refreshed origin/main snapshot before the ledger PR. | | 2026-07-14 | PR #629 / codex/eval-canary-quota-handling | 8fe1be6d0c58c60afeeb82f720b03c90ce57c2cf | review-followup | One P2 structured-error retry defect confirmed and fixed on `codex/eval-canary-structured-errors`. | GitHub connector thread inspection; `tests/eval-utils.test.ts` 14/14; focused ESLint and Prettier. | +| 2026-07-14 | claude/repo-next-steps-e53523 | 3f7d6d76f597f2a0311af1052480f84b68ecc259 | launch-readiness and RAG-performance follow-up | Branch changes were already squash-merged by PR #513. Fresh review found one P1 governance inconsistency (Singapore app/worker processing omitted from the PIA's cross-border account) and three P2 follow-ups: degraded-answer SLO overcounting, a second blocking shared-cache miss diagnostic query, and stale launch/RAG backlog status. No source fix was applied in this review. | `git diff --check c3828ceb9f3812abeebc1b653361fc254dda9f5e..3f7d6d76f597f2a0311af1052480f84b68ecc259`; focused Vitest 92/92; `npm run eval:rag:offline` 36 fixtures and 59/59 contract tests; `npm run verify:cheap` passed runtime, action pins, sitemap, type scale, lint, typecheck, and 1,672 passed/1 skipped tests. Provider-backed Supabase/OpenAI, browser, release, drift, and live retrieval-quality checks were not run. | +| 2026-07-14 | codex/rag-performance-followups | 5502fd498ea2069f810795a8659f98ab3abf8c80 | release-readiness review | The local release review found no P0-P2 defect after correcting one stale PIA statement; the later hosted review follow-up is recorded below. The scoped RAG round-trip, SLO, retention, privacy, and documentation changes were ready for PR handoff. Highest residual risk is the known live hybrid-RPC latency tail; model experiments remain blocked by provider quota and legal execution remains operator/counsel work. | Rebased onto `origin/main`; runtime and full Prettier check; ESLint; TypeScript; Vitest 2,213 passed/1 skipped; Next.js production build (636 pages) plus client-bundle secret scan; offline RAG 36 fixtures and 277/277 contract tests; `git diff --check`. Live retention jobs 13/16 were already verified during this workstream. | +| 2026-07-14 | PR #632 / codex/rag-performance-followups | aa264e92c44b42fdcceeac6292011ba51169b862 | review-followup | One P2 SLO-classification defect was confirmed: the broad source-only `degraded` flag also included healthy extractive answers. Fixed by persisting and counting a separate `provider_generation_degraded` flag derived only from `generation_fallback`. | GitHub connector plus UTF-8 thread-aware review inspection; focused ESLint; 45/45 targeted Vitest tests; TypeScript; offline RAG 36 fixtures and 277/277 contract tests; `git diff --check`. | +| 2026-07-14 | PR #632 / codex/rag-performance-followups | 07cac8f4e094bf4b5ccd264feb6420fbd4e35e2b | review-followup | CodeRabbit's APP 1/5 governance contradiction was confirmed and corrected; Railway's pending DPA and Singapore processor basis are now explicit checklist/status items. The optional cache-miss granularity suggestion was not adopted because it would undo the reviewed single-read cold-miss design or require a broader schema/RPC change. | GitHub review-thread inspection; focused Prettier and `git diff --check` passed. Local production-readiness inspection reported the expected missing provider env; the Railway production-context check had already passed on this workstream. | +| 2026-07-14 | PR #632 / codex/rag-performance-followups | 0c792b8c10516c85d18832fc96562dd70a671b5f | review-followup | One P1 privacy-disclosure defect was confirmed: OpenAI retrieval embedding can receive query text even when answer synthesis remains source-only. Fixed the PIA and public `/privacy` copy to distinguish retrieval-embedding egress from model-backed synthesis egress, with focused copy coverage. | GitHub connector review-thread inspection; focused privacy UI test, ESLint, TypeScript, Prettier, production-readiness, and `git diff --check`. | +| 2026-07-14 | PR #632 / codex/rag-performance-followups | a2eb6db0efbef983e1b3242261d5cc6b2b9d839d | review-followup | One late P2 rollout-compatibility defect was confirmed: the provider-fallback SLO would omit recent rows written before `provider_generation_degraded` existed. Fixed the count predicate to include the new flag or legacy `generation_fallback:` reasons while continuing to exclude intentional extractive routes. | GitHub connector review-thread inspection; focused answer-SLO test, ESLint, TypeScript, Prettier, and `git diff --check`. | diff --git a/docs/deployment-architecture.md b/docs/deployment-architecture.md index 656ba72411..24391288cd 100644 --- a/docs/deployment-architecture.md +++ b/docs/deployment-architecture.md @@ -61,11 +61,13 @@ properties made it the pragmatic core platform: **The trade Railway forces — and why it was accepted:** Railway has **no Australian region** (its regions are US West, US East, Amsterdam, Singapore), while Supabase is in Sydney. The app therefore pays a cross-region hop to the -database (quantified in §2.1). The penalty is real but bounded, falls almost -entirely on _novel_ (cache-miss) answers, is dwarfed by OpenAI generation time, -and is fully addressable within Railway (a Singapore read replica — §2.1). Given -that shipping a real environment was the priority, a live-in-Singapore deployment -beat an indefinitely blocked "optimal" one. +database (quantified in §2.1). Warm responses show that this hop can be bounded, +but 2026-07-14 cold probes also exposed database-execution outliers far larger +than network RTT. A Singapore replica would copy those query plans rather than +fix them, so the current first lever is RPC/query-plan work and, if needed, a +small primary-compute comparison. Given that shipping a real environment was the +priority, a live-in-Singapore deployment beat an indefinitely blocked "optimal" +one. ### Why a long-lived container and not serverless (Vercel et al.) @@ -116,6 +118,15 @@ inherent per RPC. | Authenticated PostgREST round-trip | Real per-RPC cost the app pays | **~145 ms best, ~340 ms typical** | | Production novel answer | `supabase_rpc_latency_ms` (retrieval, 3 query variants) | **~4.4 s**; total ~25 s incl. ~19 s OpenAI generation | +**Fresh production comparison, 2026-07-14:** one warmed repeat reported +`supabase_rpc_latency_ms=0`, while two cold synthesis probes reported +**48.5–49.4 s** of Supabase RPC time and **51–53 s** total. A six-case approved +live-database retrieval run on the current local code preserved perfect fixture +recall/hit-rate with **1.8 s median / 47.3 s p90** latency. Database statistics +also show large temporary-file I/O in the slow hybrid RPC families. These +outliers supersede the earlier assumption that generation or cross-region RTT is +always the dominant latency. + **What multiplies, what doesn't.** Retrieval fans out _wide_ but the RPCs within a stage run in parallel (`Promise.all`), so fan-out width costs ~1×RTT, not N×. What multiplies RTT is the sequential **depth** — ~5–8 serial DB round-trips on a @@ -134,17 +145,22 @@ shared Postgres cache tier still pay ~one origin round-trip. - keep-alive connection reuse (undici default) removes repeated client→edge handshakes on the hot path. -**Mitigations if the penalty starts to bite (not yet needed):** +**Mitigations, in current priority order:** -1. **Supabase read replica in Singapore** (biggest win). Co-locating a read - replica with Railway serves the read-heavy retrieval RPCs locally (~5 ms), - collapsing the penalty to near-parity with full co-location. Writes (telemetry, - cache) stay on the Sydney primary and are off the answer critical path. Cost: - the paid replica add-on **plus** app work to route reads to the replica - endpoint. This is the recommended first lever. +1. **Profile and optimise the slow hybrid RPC plans**, then compare the Sydney + primary on Micro versus Small compute if execution remains resource-bound. + The live primary currently exposes the Micro connection ceiling (60 direct + connections) and a small `work_mem`; scaling the primary tests query-memory + headroom without adding read routing or replica staleness. 2. **Reduce sequential DB depth** in the answer path (batch the cache-version + shared-cache probes, collapse hydration round-trips). App change; measure `latencyTimings.supabase_rpc_latency_ms` before/after. +3. **Reconsider a Singapore read replica only after the query plans are fast and + network time is again material.** Supabase currently requires at least Small + compute for read replicas; a same-size replica is asynchronous/read-only and + adds compute/storage cost plus read-routing and freshness validation. It is + not justified by the current evidence because it would reproduce the observed + execution outliers. The OpenAI leg is region-agnostic: OpenAI is US-hosted, so app→OpenAI RTT is comparable (~200 ms) from Singapore or Sydney and does not favour either host. diff --git a/docs/launch-operator-runbook.md b/docs/launch-operator-runbook.md index 335d59c6d5..5f34ce79bc 100644 --- a/docs/launch-operator-runbook.md +++ b/docs/launch-operator-runbook.md @@ -5,9 +5,14 @@ runs automatically — every **⏸ PAUSE** is a provider-touching action (Supabase / Railway / OpenAI / GitHub) that needs your explicit go-ahead, per the AGENTS.md provider boundary. +**Current-state note (2026-07-14):** production app and worker deployment is already recorded as live. +Use this as a verification/release sequence, not evidence that staging, migrations, or first deployment +are still pending. Confirm current provider state before repeating any historical action. + Host note: production app + worker run on **Railway** (user directive 2026-07-12), not Fly. The image is host-agnostic. Railway has no Sydney region (closest Singapore); data at rest stays in Supabase Sydney, -so this is a latency/SLO tradeoff only — confirm answer-p95 in the staging soak (step 4). +so this creates both a latency/SLO tradeoff and overseas processing that must be covered by the approved +privacy/processor record. Confirm answer-p95 in the staging soak (step 4). Legend: **⏸ PAUSE** = provider action, needs your approval · **✅ verify** = check to run after. @@ -17,9 +22,9 @@ Legend: **⏸ PAUSE** = provider action, needs your approval · **✅ verify** = ```text 0. Pre-flight identity check -1. Apply pending live migrations (July-8 batch + PIA-4 + drift-codify) [Supabase] -2. Run the full release gate [live keys] -3. Provision staging + seed [Supabase + Railway] +1. Confirm completed migrations; apply only any explicitly unresolved control [Supabase] +2. Run the full release gate before the next release [live keys] +3. Provision staging + seed, only if still absent [Supabase + Railway] 4. Staging soak + rollback rehearsal [Railway] 5. Production deploy [Railway] 6. Post-deploy: worker, registry seed, auth conn cap, observability wiring @@ -35,26 +40,27 @@ npx supabase migration list --linked npm run reindex:health # note jobs_pending / jobs_processing (needed for step 1 R17) ``` -## 1. Apply pending live migrations 🧑 Supabase +## 1. Confirm migration state; apply only unresolved controls 🧑 Supabase + +The July-8 ingestion/tenancy batch and the retrieval drift-codification work are recorded as applied and +verified on 2026-07-13. Their detailed procedures remain for staging/disaster recovery; do **not** reapply +them merely because they appear below. First compare linked migration history and verify the remaining +PIA-4 retention migrations. **Do not redeploy the worker until `20260708130000` is confirmed live.** -Detailed runbook: [operator-apply-july8-batch.md](operator-apply-july8-batch.md). Apply **in this order** -when the ingestion queue is quiet. **Do not redeploy the worker until step `20260708130000` is live.** +| Group | Migration/control | Recorded status | Operator action | +| ----- | ------------------------------------------------------------------------------------------------ | --------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | +| a–f | July-8 ingestion/tenancy batch in [operator-apply-july8-batch.md](operator-apply-july8-batch.md) | **Verified live 2026-07-13** | Verify only; redeploy the worker if that recorded remaining action is still open. | +| g | `20260708120000_rag_query_misses_retention` | **Applied and verified live 2026-07-14** | Job 13 is active with the 90-day retention window. | +| h | `20260713201542_consolidate_rag_response_cache_retention` | **Applied and verified live 2026-07-14** | Job 16 is active and bounded; obsolete duplicate job confirmed absent. | +| i | Retrieval RPC forward-codification (`20260713062107`…`20260713062139`) | **Applied and drift/readiness verified 2026-07-13** | Verify only; see [forward-codify-retrieval-rpcs-workorder.md](forward-codify-retrieval-rpcs-workorder.md). | -| # | Migration | Note | -| --- | ----------------------------------------------------- | ------------------------------------------- | -| a | `20260708140000_drop_ingestion_job_stages_job_id_fk` | no-op on live | -| b | `20260708130000_ingestion_concurrency_rpc_hardening` | **worker-redeploy blocker** | -| c | `20260708150000_ensure_retrieval_owner_matches` | helper before fail-closed | -| d | `20260708160001_retrieval_owner_matches_fail_closed` | tenancy fail-closed (#409) | -| e | `20260708310000_r5_document_metadata_merge` | R5 deep-merge (#408) | -| f | `20260708170000_ingestion_jobs_one_open_per_document` | R17 — approved manual `CONCURRENTLY` path | -| g | `20260708120000_rag_query_misses_retention` | **PIA-4** purge cron | -| h | `` | **only after task 1.2 lands** — see step 1b | +**⏸ PAUSE:** if and only if linked history shows migration `20260708120000` or `20260713201542` absent, +apply the reviewed committed migration through the normal guarded workflow. Do not use this status +reconciliation as authority to replay the verified July-8 or forward-codification groups. -**⏸ PAUSE:** apply via `supabase db push` (queue quiet) or the R17 manual `CREATE UNIQUE INDEX CONCURRENTLY` -path in the July-8 doc. R17 manual path is an approved exception to the live-change guardrail; record -the migration history entry and reconcile schema.sql after manual execution to prevent untracked drift. -R17 uses its own version so history/repair can't collide with `20260708160001`. +After applying, query `cron.job` and expect exactly these active retention jobs: `purge-rag-query-misses` +at `45 3 * * *` and `purge-rag-response-cache` at `15 * * * *`. The obsolete +`purge-expired-rag-response-cache` name must be absent. **✅ verify:** @@ -65,20 +71,17 @@ npm run check:indexing # search_schema_health() ok npm run eval:retrieval:quality # must stay 36/36 (retrieval-affecting: step d + drift-codify) ``` -### 1b. Drift-codify apply (task 1.2) +### 1b. Drift-codify status (task 1.2) — complete -The forward-codify migration (live-diverged `match_document_chunks` `hnsw.ef_search=100` wrapper + `*_text` -multi-strategy bodies) is authored + validated with normalized fingerprint comparison vs a Docker replay -before it reaches you, so its apply is an **idempotent no-op on live**. **This step is blocked until the -migration artifact is committed and execution-time live fingerprint recapture is completed.** Before -applying, recapture live fingerprints using the exact committed capture query and compare normalized md5s -against the committed table. Abort on mismatch. Apply as step 1h only after verification, then re-run -`check:drift` + `eval:retrieval:quality` (36/36). Background: -[database-drift-detection.md](database-drift-detection.md). +The reviewed forward-codify migrations were applied on 2026-07-13 after scratch replay and fingerprint +validation. `check:drift` passed and production readiness reported READY. The authoritative evidence and +historical replay procedure are in +[forward-codify-retrieval-rpcs-workorder.md](forward-codify-retrieval-rpcs-workorder.md). No further apply +is pending unless a new drift check identifies a new, separately reviewed difference. ## 2. Full release gate 🧑 live keys -Clears the accumulated verification debt (universal search, cross-mode links, rag.ts decomp). +Run before the next release when provider-backed verification is explicitly approved. **⏸ PAUSE** (bounded OpenAI spend): @@ -161,6 +164,6 @@ no scale-to-zero, health `/api/health`. I'll prep the Railway service config via ## Standing guardrails - Never raw-SQL against live — committed migration + `schema.sql` reconciliation only. -- Worker redeploy is blocked until `20260708130000` is live. +- Never redeploy a worker image that expects the hardened completion RPC until `20260708130000` is confirmed live. - Any retrieval/ranking change re-runs `eval:retrieval:quality` 36/36 before it ships. - Each environment gets separate service-role + OpenAI keys (per-env blast radius). diff --git a/docs/observability-slos.md b/docs/observability-slos.md index 925dac59ec..b1a3889fc3 100644 --- a/docs/observability-slos.md +++ b/docs/observability-slos.md @@ -118,7 +118,12 @@ means the OpenAI path is broken while users still get 200s. - **Warn:** > 20 % over 1 h. **Page:** > 50 % over 1 h (generation is effectively down). -Measure via `metadata->>'fallback_reason'` / `answer_model_demoted`. +Measure provider-generation degradation via the explicit +`metadata->>'provider_generation_degraded' = 'true'` flag. During rollout, the +probe also counts legacy `fallback_reason ILIKE '%generation_fallback:%'` rows so +recent pre-flag provider failures remain visible until they age out. Keep `degraded` for +the broader source-only UI state and `fallback_reason` as diagnostic detail; +neither is narrow enough for provider health on its own. ## 3. Nightly production eval canary @@ -177,7 +182,8 @@ header (same operator gate as the Supabase probe) — returns two counter blocks - **`slo`** — `answerSloSnapshot` (`src/lib/observability/answer-slo.ts`) counts `rag_queries` over the trailing `windowMinutes` (60) and reports `hybridRpcErrorQueries` / `hybridRpcErrorRate` (the §2 silent-RPC-death guard) - and `degradedQueries` / `degradedRate` (the source-only/fallback guard). These + and `degradedQueries` / `degradedRate` (the explicit provider-generation + fallback guard, excluding intentional extractive and unsupported/source-gap outcomes). These are windowed rates straight from the persisted telemetry. - **`cache`** — `cacheMetricsSnapshot` (`src/lib/observability/cache-metrics.ts`) diff --git a/docs/openai-cross-border-basis.md b/docs/openai-cross-border-basis.md index af236ee6f5..b7ffdced55 100644 --- a/docs/openai-cross-border-basis.md +++ b/docs/openai-cross-border-basis.md @@ -1,9 +1,9 @@ # Cross-border disclosure basis — OpenAI (PIA-1) -**Status:** Decision-ready — awaiting the operator/legal step · **Date:** 2026-07-13 +**Status:** Provider review refreshed — operator approved pursuit; legal execution pending · **Date:** 2026-07-14 **Owner of the open step:** account holder for `OPENAI_API_KEY` + privacy adviser -**Closes:** the contractual half of **PIA-1** in [docs/privacy-impact-assessment.md](privacy-impact-assessment.md) §10. -**Companion:** the `/privacy` collection notice ([src/app/privacy/page.tsx](../src/app/privacy/page.tsx)) and composer reminder ([src/lib/ui-copy.ts](../src/lib/ui-copy.ts)) satisfy the APP 5 / APP 1 half. +**Tracks:** the OpenAI contractual portion of **PIA-1** in [docs/privacy-impact-assessment.md](privacy-impact-assessment.md) §10; Railway's companion contract is tracked below. +**Companion:** the `/privacy` collection notice ([src/app/privacy/page.tsx](../src/app/privacy/page.tsx)) and composer reminder ([src/lib/ui-copy.ts](../src/lib/ui-copy.ts)) ship as draft APP 5 / APP 1 controls pending governance approval under **PIA-5**. > **Not legal advice.** This records the current, verifiable facts about OpenAI's data-handling > terms and maps them to APP 8 so a qualified privacy adviser can sign off the cross-border basis. @@ -14,11 +14,13 @@ ## 1. Why this exists -The app's only cross-border flow is the query text + retrieved excerpts sent to OpenAI in the United -States for embedding and answer synthesis (PIA §3–4; verified still true in code — +This document covers the model-provider leg: query text + retrieved excerpts sent to OpenAI in the +United States for embedding and answer synthesis (PIA §3–4; verified still true in code — [src/lib/openai.ts:75-79](../src/lib/openai.ts) builds a plain `new OpenAI({ apiKey, timeout, maxRetries })` with no `baseURL`/ZDR header, `store:false` by default, and `prompt_cache_retention` forced to `"24h"` -for gpt-5.5 at [openai.ts:174](../src/lib/openai.ts)). +for gpt-5.5 at [openai.ts:174](../src/lib/openai.ts)). Railway application and worker processing in +Singapore is the separate overseas processor leg recorded in the PIA; the APP 8 record must cover +both providers. Two obligations attach to that flow: @@ -28,7 +30,8 @@ Two obligations attach to that flow: recipient's acts unless an APP 8.2 exception applies. Health/mental-health data is _sensitive information_ — the highest-protection category — so this is the launch-critical item. - **APP 5 (notification).** Individuals must be told their information is disclosed overseas. - **Already shipped** in the `/privacy` page and composer notice (see §7). + Draft wording is **already shipped** in the `/privacy` page and composer notice; governance approval + remains open under **PIA-5** (see §7). The code-side controls cannot _by themselves_ discharge APP 8 — the "reasonable steps" are largely **contractual**. That contract is the open step this document tracks. @@ -93,38 +96,50 @@ mitigated. Record the answer in the status block. ## 7. Consistency with the shipped user-facing notice -The `/privacy` page and composer notice already tell users: data is stored in **Sydney**; question -text + excerpts go to **OpenAI in the US** ("the only point where data leaves Australia"); OpenAI is -**asked not to retain** requests (`store:false`); retention is 30d/90d. This document must stay -consistent with those claims. +The `/privacy` page and composer notice tell users that durable database/storage data is in +**Sydney**, Railway processes requests in **Singapore**, and question text + excerpts go to +**OpenAI in the US**. OpenAI is asked not to store response objects (`store:false`), while separate +provider retention and local 30d/90d retention are disclosed. This document must stay consistent +with those claims. -- **Merge status:** the APP-5 half is **live on `main`** — `src/app/privacy/page.tsx` and the composer - notice landed via **PR #513** (`eeb2340ad`). So APP 5/1 is met; only the APP 8 contractual basis below - remains. -- **Follow-up:** if **Australia data residency** is enabled, the `/privacy` copy ("the only point where - data leaves Australia") stays accurate for _processing_, but the "where stored" section can be - strengthened to note US/AU storage — update copy if residency is adopted. +- **Merge status:** the draft APP-5/1 controls are **live on `main`** — `src/app/privacy/page.tsx` and + the composer notice landed via **PR #513** (`eeb2340ad`). Their final governance approval remains + open under **PIA-5**, alongside the APP 8 contractual basis below. +- **Follow-up:** if **Australia data residency** is enabled, update the "where stored" section to + describe the selected OpenAI storage region without implying that inference is processed in + Australia. ## 8. Operator action checklist Actions **1–3 must be performed by the account holder** in OpenAI's dashboard/legal process — they involve accepting agreements and changing account settings, which an automated agent must not do. +The operator approved pursuing these steps on 2026-07-14. That approval authorises evaluation and +outreach; it is not a legal signature and does not prove that DPA, ZDR, or residency settings are in +force. The production project also returned quota exhaustion during the approved model experiment, +so endpoint/model comparisons are paused until quota is restored. ZDR/residency status was not +available through the inspected API surface and remains to be confirmed in the provider account or +in writing. + - [ ] **1. Execute the OpenAI DPA** for the org behind the production `OPENAI_API_KEY` → [openai.com/policies/data-processing-addendum](https://openai.com/policies/data-processing-addendum/). Store the countersigned copy; record version + date below. -- [ ] **2. Apply for ZDR** on the production project via the OpenAI account/sales team. Confirm it +- [ ] **2. Execute Railway's DPA** with the legal entity and authorised signer. Retain the executed + copy and record the Singapore processor/sub-processor basis. +- [ ] **3. Apply for ZDR** on the production project via the OpenAI account/sales team. Confirm it covers `/v1/responses` + `/v1/embeddings`. Record project id + approval date. -- [ ] **3. Decide on Australia data residency** (new Project + country selection; sales-gated). +- [ ] **4. Decide on Australia data residency** (new Project + country selection; sales-gated). Record region + date, or record an explicit decision not to adopt it and why. -- [ ] **4. Confirm the ZDR ↔ prompt-cache behaviour** in writing (§6); record the answer. -- [ ] **5. Review the sub-processor list** for anything counsel should note in the APP 8 chain. -- [ ] **6. Legal sign-off** that the §5 package satisfies APP 8 for sensitive health information. -- [ ] **7. Keep `/privacy` copy in sync** if AU residency is adopted (§7) — note US/AU storage. -- [ ] **8. Code follow-ups** once the above land (§9), if adopted. +- [ ] **5. Confirm the ZDR ↔ prompt-cache behaviour** in writing (§6); record the answer. +- [ ] **6. Review both providers' sub-processor lists** for anything counsel should note in the APP 8 chain. +- [ ] **7. Legal sign-off** that the §5 package satisfies APP 8 for sensitive health information and + approves the shipped draft APP 5/1 wording under PIA-5. +- [ ] **8. Keep `/privacy` copy in sync** if AU residency is adopted (§7) — note US/AU storage. +- [ ] **9. Code follow-ups** once the above land (§9), if adopted. -> APP 5/1 (the collection notice + `/privacy` page) is **already done** — live on `main` via PR #513. -> This checklist covers only the remaining **APP 8** contractual basis. +> Draft APP 5/1 controls (the collection notice + `/privacy` page) are live on `main` via PR #513. +> Final governance approval remains open under **PIA-5**; this checklist also tracks the remaining +> **APP 8** contractual basis for both overseas providers. ### Status record — fill in as steps complete @@ -132,13 +147,20 @@ involve accepting agreements and changing account settings, which an automated a | ----------------------------------- | ------------------------- | ---- | -------- | | OpenAI org / production project id | _tbd_ | | | | DPA executed (version) | _no (v.010126 available)_ | | | +| Railway DPA executed (version) | _no_ | | | +| Railway Singapore processor basis | _pending_ | | | | ZDR approved (project) | _no_ | | | | ZDR covers /responses + /embeddings | _tbd_ | | | | ZDR zeroes prompt cache? (§6) | _tbd_ | | | | Australia data residency | _not enabled_ | | | | No-training confirmed in contract | _API default_ | | | +| APP 5/1 notice governance approval | _pending (PIA-5)_ | | | | Counsel sign-off (APP 8) | _pending_ | | | +Railway's companion contract step is also pending: complete Railway's DPA with the legal entity and +authorised signer, retain the executed copy, and record the Singapore processor/sub-processor basis +in the PIA. No automated action in this review accepted either provider's terms. + ## 9. Code follow-ups triggered by the outcome These touch the OpenAI request path — do them **only after** the legal decision, and treat them as @@ -160,5 +182,6 @@ provider-path changes (confirm before running against live). - OpenAI — [Data Processing Addendum](https://openai.com/policies/data-processing-addendum/) · [PDF v.010126](https://cdn.openai.com/pdf/openai-data-processing-addendum.pdf) - OpenAI — [Sub-processor list](https://openai.com/policies/sub-processor-list/) - OpenAI — [Data residency for the OpenAI API](https://help.openai.com/en/articles/10503543-data-residency-for-the-openai-api) · [Expanding data residency worldwide](https://openai.com/index/expanding-data-residency-access-to-business-customers-worldwide/) +- Railway — [Data Processing Addendum](https://railway.com/legal/dpa) · [Trust Center](https://trust.railway.com/) - OAIC — Australian Privacy Principles (APP 8 cross-border disclosure; s16C accountability), _Privacy Act 1988_ (Cth) - Internal — [Privacy Impact Assessment](privacy-impact-assessment.md) (PIA-1, PIA-6) diff --git a/docs/privacy-impact-assessment.md b/docs/privacy-impact-assessment.md index 7f5969d503..c9dfb74b23 100644 --- a/docs/privacy-impact-assessment.md +++ b/docs/privacy-impact-assessment.md @@ -1,7 +1,7 @@ # Privacy Impact Assessment — Clinical KB Database -**Status:** Draft for review · **Date:** 2026-07-06 · **Branch:** `claude/privacy-tenancy-review` -**Scope:** Clinical data flows through the Clinical KB app (Next.js + Supabase + OpenAI), the live Supabase project `Clinical KB Database` (`sjrfecxgysukkwxsowpy`), and the WA private-clinical deployment context. +**Status:** Draft for review · **Date:** 2026-07-06 · **Revised:** 2026-07-14 +**Scope:** Clinical data flows through the Clinical KB app (Next.js on Railway Singapore + Supabase Sydney + OpenAI), the live Supabase project `Clinical KB Database` (`sjrfecxgysukkwxsowpy`), and the WA private-clinical deployment context. **Author:** Automated code-level assessment (multi-agent audit of `src/app/api/**`, `src/lib/*`, `supabase/schema.sql`, `supabase/migrations/**`), cross-checked against the live database. > **This is not legal advice.** It is a technical privacy assessment written to be handed to a @@ -19,7 +19,10 @@ generation. It is **not** a patient record system and, by design, does not ask f The dominant privacy risk is therefore **incidental PHI**: a clinician will inevitably type patient details into a free-text query ("42yo F on clozapine 400mg with rising WCC, next step?"). That query -text is (a) sent to OpenAI in the United States, and (b) written to log tables in Supabase. A secondary +is processed by the Railway application tier in Singapore and can be sent to OpenAI in the United +States for retrieval embedding even when the final answer is source-only. When model-backed answer +synthesis is used, the query and selected excerpts are sent again. The query is hash-redacted before +it is written to log tables in Supabase. A secondary risk is PHI inside **uploaded documents** if users upload anything other than published reference material. @@ -42,28 +45,28 @@ material. **Top gaps (full register in §10):** -| ID | Risk | One-line | -| ----- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| PIA-1 | High | Cross-border disclosure to OpenAI (US) has no code-visible DPA/ZDR. A draft provider disclosure now ships in-product, but its wording lacks governance approval (APP 8, APP 5). | -| PIA-2 | High | Production now fails closed without `RAG_QUERY_HASH_SECRET`; the operator must still place the secret in the deploy host. | -| PIA-3 | Mitigated | Generated answer text is omitted from `rag_queries` by default. `RAG_PERSIST_ANSWER_TEXT=true` is explicit opt-in and blocked by production readiness. | -| PIA-4 | Medium | `rag_query_misses` has **no retention/purge job** (only `rag_queries` and `rag_retrieval_logs` do). | -| PIA-5 | Medium | Draft point-of-entry collection notices and a `/privacy` data-processing page ship, but no governance-approved final privacy policy exists (APP 1, APP 5). | -| PIA-6 | Low-Med | OpenAI **prompt-cache retention is forced to 24h** for gpt-5.5 regardless of config — query + retrieved excerpts persist ≤24h at OpenAI. | -| PIA-7 | Low | `RAG_PERSIST_RAW_QUERY_TEXT=true` would store raw PHI query text with no secondary safeguard beyond the 30-day purge. | +| ID | Risk | One-line | +| ----- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | +| PIA-1 | High | Overseas processing occurs in Railway Singapore and OpenAI US; the applicable processor/APP 8 basis and final notice wording require governance approval. | +| PIA-2 | High | Production now fails closed without `RAG_QUERY_HASH_SECRET`; the operator must still place the secret in the deploy host. | +| PIA-3 | Mitigated | Generated answer text is omitted from `rag_queries` by default. `RAG_PERSIST_ANSWER_TEXT=true` is explicit opt-in and blocked by production readiness. | +| PIA-4 | Mitigated | Query-miss and bounded response-cache purges were verified active live on 2026-07-14; the duplicate unbounded cache job was removed. | +| PIA-5 | Medium | Draft point-of-entry collection notices and a `/privacy` data-processing page ship, but no governance-approved final privacy policy exists (APP 1, APP 5). | +| PIA-6 | Low-Med | OpenAI **prompt-cache retention is forced to 24h** for gpt-5.5 regardless of config — query + retrieved excerpts persist ≤24h at OpenAI. | +| PIA-7 | Low | `RAG_PERSIST_RAW_QUERY_TEXT=true` would store raw PHI query text with no secondary safeguard beyond the 30-day purge. | --- ## 2. System overview and data classification -| Data category | Where it lives | Sensitivity | Notes | -| ------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------- | -| Clinical reference corpus (documents, chunks, embeddings, images, tables) | Supabase (Sydney) + storage buckets | Low–Medium | Published guidelines are not PHI; **uploaded** docs _could_ contain PHI. | -| Free-text clinical queries | Transient in request; hashed into `rag_queries` / `rag_query_misses` / `rag_retrieval_logs`; sent to OpenAI (US) | **High (potential PHI)** | The primary incidental-PHI vector. | -| Generated answers | `rag_queries.answer` (not persisted unless `RAG_PERSIST_ANSWER_TEXT`); `rag_response_cache.payload` (read-TTL, no purge cron) | **High (derived from PHI query + corpus)** | Durable answer log dropped at rest by default (PIA-3); cache row persists until same-key overwrite (TTL gates reads only). | -| User identity | Supabase Auth (`auth.users`), `owner_id` foreign keys | Medium (PII) | Email + SSO identity; managed by Supabase Auth. | -| Audit trail | `audit_logs` | Medium | Append-only, service-role-only, retained indefinitely by design. | -| Operational telemetry | `rag_retrieval_logs`, ingestion job tables | Low–Medium | Redacted query text; per-owner. | +| Data category | Where it lives | Sensitivity | Notes | +| ------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------- | +| Clinical reference corpus (documents, chunks, embeddings, images, tables) | Supabase (Sydney) + storage buckets | Low–Medium | Published guidelines are not PHI; **uploaded** docs _could_ contain PHI. | +| Free-text clinical queries | Processed by Railway (Singapore); hashed into Supabase logs (Sydney); sent to OpenAI (US) for retrieval embedding and, when selected, answer synthesis | **High (potential PHI)** | The primary incidental-PHI vector; embedding egress can occur even when the final answer is source-only. | +| Generated answers | `rag_queries.answer` (not persisted unless `RAG_PERSIST_ANSWER_TEXT`); short-lived `rag_response_cache.payload` | **High (derived from PHI query + corpus)** | Durable answer log dropped at rest by default (PIA-3); expired cache rows have a bounded hourly purge when `pg_cron` is available. | +| User identity | Supabase Auth (`auth.users`), `owner_id` foreign keys | Medium (PII) | Email + SSO identity; managed by Supabase Auth. | +| Audit trail | `audit_logs` | Medium | Append-only, service-role-only, retained indefinitely by design. | +| Operational telemetry | `rag_retrieval_logs`, ingestion job tables | Low–Medium | Redacted query text; per-owner. | **Deployment context (from code):** the answer system prompt positions the assistant as _"an experienced psychiatrist in Perth"_ ([src/lib/rag.ts:7053](src/lib/rag.ts)) — i.e. a **WA psychiatry** @@ -80,7 +83,7 @@ The end-to-end path for a single clinician query. **Bold** nodes are where PHI c Clinician browser │ POST /api/answer { query: "", ... } ▼ -[Next.js route] src/app/api/answer/route.ts:70 +[Next.js route — Railway Singapore] src/app/api/answer/route.ts:70 │ • auth resolved → access.ownerId (or undefined for anon/public) :80 │ • rate-limit bucket "answer" :83 │ • resolveSearchScope() → owner-scoped candidate document set :93 @@ -120,8 +123,10 @@ Clinician browser ← answer + citations `rag_retrieval_logs` (all redacted via the same helpers — [src/app/api/search/route.ts:450-468, 556-559, 638-643](src/app/api/search/route.ts)). -**The two egress points that carry PHI off-app are (A) and (C) — both to OpenAI in the US.** -Everything in Supabase stays in Sydney. +The browser request, answer pipeline, and ingestion worker are processed by Railway in Singapore. The +model egress points (A) and (C) then carry query/evidence content to OpenAI in the US. Durable Supabase +data remains in Sydney. Governance must assess both overseas processing paths rather than treating +OpenAI as the only cross-border flow. --- @@ -168,8 +173,9 @@ facts, and must be confirmed: retention) — this needs to be pinned to the specific contract, not assumed. Under **APP 8 (cross-border disclosure)**, the app operator remains accountable for OpenAI's handling -of the disclosed information unless an APP 8.2 exception applies. This is the single most important -privacy item to close before real patient use (PIA-1). +of the disclosed information unless an APP 8.2 exception applies. The corresponding processor/legal +assessment for Railway Singapore must also be recorded. Closing these overseas-processing terms is one +of the most important privacy items before real patient use (PIA-1). --- @@ -233,13 +239,13 @@ which is why the query-hash approach (not raw storage) is the right primary cont ## 6. Retention and purge -| Data | Retention | Mechanism | Live status | -| -------------------- | ---------------------- | ------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------- | -| `rag_queries` | 30 days | `purge_expired_rag_queries(30)`, `pg_cron` `purge-expired-rag-queries` @ 03:30 UTC | **Active** (jobid 11, verified live) | -| `rag_retrieval_logs` | 90 days | `pg_cron` `purge-rag-retrieval-logs` @ 03:00 UTC | **Active** (jobid 12, verified live) | -| `rag_query_misses` | **None** | — | **No purge job** — see PIA-4 | -| `rag_response_cache` | ~5 min TTL (soft) | `expires_at` filtered on read; overwritten per query | Rows expire logically; no hard purge cron (low volume, short TTL) | -| `audit_logs` | Indefinite (by design) | Documented in [migration 20260702120000](supabase/migrations/20260702120000_rag_retrieval_logs_retention.sql):8-12 | Intentional; "do not add purge without compliance review" | +| Data | Retention | Mechanism | Live status | +| -------------------- | ---------------------- | -------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- | +| `rag_queries` | 30 days | `purge_expired_rag_queries(30)`, `pg_cron` `purge-expired-rag-queries` @ 03:30 UTC | **Active** (jobid 11, verified live) | +| `rag_retrieval_logs` | 90 days | `pg_cron` `purge-rag-retrieval-logs` @ 03:00 UTC | **Active** (jobid 12, verified live) | +| `rag_query_misses` | 90 days | `purge_expired_rag_query_misses(90)`, `pg_cron` `purge-rag-query-misses` @ 03:45 UTC | **Active** (jobid 13, verified live 2026-07-14) | +| `rag_response_cache` | ~5 min read TTL | `expires_at` filtered on read; `purge_expired_rag_response_cache(1000)`, hourly `pg_cron` `purge-rag-response-cache` | **Active** (jobid 16, verified live 2026-07-14); obsolete unbounded job removed | +| `audit_logs` | Indefinite (by design) | Documented in [migration 20260702120000](supabase/migrations/20260702120000_rag_retrieval_logs_retention.sql):8-12 | Intentional; "do not add purge without compliance review" | **Verification (live `cron.job` query, 2026-07-06):** @@ -248,11 +254,15 @@ jobid 11 purge-expired-rag-queries 30 3 * * * active=true select public.pu jobid 12 purge-rag-retrieval-logs 0 3 * * * active=true delete from public.rag_retrieval_logs where created_at < now() - interval '90 days'; ``` -So the answer to "_is anything scheduled?_" is **yes** — the two query-log purges are live and active. -The retention story is sound **except**: +So the answer to "_is anything scheduled?_" is **yes** for the two jobs verified on 2026-07-06. Since +that verification, migration `20260708120000_rag_query_misses_retention.sql` added a 90-day query-miss +purge, and migration `20260713201542_consolidate_rag_response_cache_retention.sql` consolidated two +cache purge jobs onto the existing bounded hourly purge. The remaining retention work is: -- **PIA-4:** `rag_query_misses` (which stores the same hashed-query + metadata as `rag_queries`) has - **no** purge job — it accumulates indefinitely. It should get a matching 30–90 day cron. +- **PIA-4 verification:** production was verified on 2026-07-14: migration `20260708120000` runs + `purge-rag-query-misses` as job 13, and migration `20260713201542` runs the bounded + `purge-rag-response-cache` as job 16. The obsolete `purge-expired-rag-response-cache` job is absent. + Repeat this check for any secondary environment that retains real data. - The purge functions are installed conditionally (`if to_regnamespace('cron') is null then return`, [migration 20260629060603](supabase/migrations/20260629060603_rag_queries_retention.sql):27-43) — fine on live (pg_cron present) but **preview/branch databases silently skip scheduling**. Not a @@ -265,13 +275,19 @@ The retention story is sound **except**: - **Supabase project region: `ap-southeast-2` (AWS Asia Pacific, Sydney).** Confirmed via the Supabase management API for project `sjrfecxgysukkwxsowpy`. All Postgres data (documents, chunks, embeddings, logs, auth) and both storage buckets are **onshore in Australia**. This is a strong position for WA - clinical use and directly supports APP 8 / APP 11 expectations for health information. + clinical use and directly supports APP 11 expectations for health information. +- **Railway application and worker: Singapore.** Browser questions, retrieved evidence, generated + answers, and ingestion material are processed by the Railway services before reads/writes reach + Supabase Sydney. The operator must record the applicable processor, contract, and APP 8 assessment; + this technical PIA does not decide the legal classification. - **OpenAI: United States.** Query text + retrieved excerpts are disclosed to `api.openai.com` (no - regional/EU endpoint or ZDR configured in code). This is the **only** cross-border flow, and it is - the crux of the APP 8 assessment (PIA-1). + regional endpoint or ZDR configured in code). This is the second overseas processing path and remains + a central part of the APP 8 assessment (PIA-1). -**Net:** data _at rest_ is Australian; data _in transit for inference_ crosses to the US. A privacy -notice must disclose the OpenAI disclosure and its purpose. +**Net:** durable Supabase data is Australian; application/worker processing occurs in Singapore; and +OpenAI retrieval embedding plus model-backed inference occur in the US. Embedding egress can happen +even when the final answer is source-only. The approved privacy notice and contractual record must +cover both overseas paths and their purposes. --- @@ -308,38 +324,42 @@ the operative framework for a WA private clinician. (The WA _Privacy and Respons Act 2024_ targets WA **public-sector** entities and may apply to public-health deployments — confirm with counsel if this is deployed inside a WA Health service.) -| APP | Obligation | Status in this app | Gap | -| ----------------------------------------- | ---------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------- | -| **APP 1** — open & transparent management | Have a clear, up-to-date APP privacy policy | A draft `/privacy` data-processing page ships, but it is explicitly governance-review-required and is not represented as the final approved APP privacy policy | PIA-5 | -| **APP 3** — collection of sensitive info | Collect health info only with consent + where reasonably necessary | App does not solicit PHI; incidental entry remains possible. “Do not enter patient-identifiable information” notices now appear beside query/upload controls, but no governance-approved consent framework is claimed | PIA-5 | -| **APP 5** — notification of collection | Tell individuals what's collected & disclosed (incl. overseas) | Draft point-of-entry notices and the `/privacy` page disclose data processing and provider use; final wording and legal/governance approval remain outstanding | PIA-1, PIA-5 | -| **APP 6** — use/disclosure | Use only for the primary purpose or a permitted secondary purpose | Query used for answer generation (primary). Log retention = quality/eval (secondary) — defensible but should be documented | PIA-5 | -| **APP 8** — cross-border disclosure | Discloser stays accountable for the overseas recipient unless an exception applies | Disclosure to OpenAI (US); no code-visible DPA/ZDR; accountability unclear | **PIA-1** | -| **APP 11** — security & destruction | Reasonable security; destroy/de-identify when no longer needed | Strong: Sydney residency, RLS, private storage, query hashing, default-null answer logs, 30/90-day purge. Remaining gaps are operator secret placement, exceptional answer-persistence governance, `rag_response_cache` expiry cleanup, and PIA-4 (misses never purged) | PIA-2/4 | -| **NDB scheme** (Pt IIIC) | Notify OAIC + individuals of eligible breaches of health info | No documented breach-response runbook tied to these tables | Recommend adding | +| APP | Obligation | Status in this app | Gap | +| ----------------------------------------- | ---------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------- | +| **APP 1** — open & transparent management | Have a clear, up-to-date APP privacy policy | A draft `/privacy` data-processing page ships, but it is explicitly governance-review-required and is not represented as the final approved APP privacy policy | PIA-5 | +| **APP 3** — collection of sensitive info | Collect health info only with consent + where reasonably necessary | App does not solicit PHI; incidental entry remains possible. “Do not enter patient-identifiable information” notices now appear beside query/upload controls, but no governance-approved consent framework is claimed | PIA-5 | +| **APP 5** — notification of collection | Tell individuals what's collected & disclosed (incl. overseas) | Draft point-of-entry notices and the `/privacy` page disclose Singapore application processing and model-provider use; final wording and legal/governance approval remain outstanding | PIA-1, PIA-5 | +| **APP 6** — use/disclosure | Use only for the primary purpose or a permitted secondary purpose | Query used for answer generation (primary). Log retention = quality/eval (secondary) — defensible but should be documented | PIA-5 | +| **APP 8** — cross-border disclosure | Discloser stays accountable for the overseas recipient unless an exception applies | Railway processing in Singapore and OpenAI processing in the US require documented contractual/legal assessment; OpenAI has no code-visible DPA/ZDR | **PIA-1** | +| **APP 11** — security & destruction | Reasonable security; destroy/de-identify when no longer needed | Strong: Sydney data residency, RLS, private storage, query hashing, default-null answer logs, and live-verified query/log/cache purges. Remaining gaps are operator secret placement, secondary-environment schedule parity, and exceptional answer-persistence governance | PIA-2/4 | +| **NDB scheme** (Pt IIIC) | Notify OAIC + individuals of eligible breaches of health info | No documented breach-response runbook tied to these tables | Recommend adding | **Overall:** the _engineering_ controls for data-at-rest are strong and largely APP-11-aligned. The material shortfalls are **governance/contractual** (APP 8 cross-border terms and final approval of the -draft APP 1/5 policy/notice wording) plus the -**hardening** items of operator HMAC-secret placement and query-miss/cache retention. Answer prose is omitted by default; -enabling its persistence is an exceptional, non-production mode requiring governance approval. Anonymous answer caching -is disabled. The tenancy review found **zero** confirmed cross-tenant leaks; the remaining items are compliance-posture -and PHI-minimisation gaps. +draft APP 1/5 policy/notice wording) plus the **hardening** items of operator HMAC-secret placement and +retention-schedule parity in any secondary environment that stores real data. Answer prose is omitted by +default; enabling its persistence is an exceptional, non-production mode requiring governance approval. +Anonymous answer caching is disabled. The tenancy review found **zero** confirmed cross-tenant leaks; the +remaining items are compliance-posture and PHI-minimisation gaps. --- ## 10. Gap register (ranked by risk) -### PIA-1 — Cross-border disclosure to OpenAI lacks visible DPA/ZDR + notice **(High)** +### PIA-1 — Overseas Railway/OpenAI processing needs an approved contractual basis and notice **(High)** -- **Risk:** Health/PHI in queries and excerpts is disclosed to OpenAI (US) with no code-visible - contractual data-processing terms. A draft in-product provider disclosure now exists, reducing the +- **Risk:** Health/PHI in requests, queries, excerpts, and ingestion material is processed by Railway + in Singapore. Query text can reach OpenAI in the US for retrieval embedding even when the final + answer is source-only; model-backed synthesis additionally sends the query and selected excerpts. + OpenAI has no code-visible contractual data-processing terms. A draft in-product provider disclosure now exists, reducing the point-of-entry visibility gap, but it is not governance-approved legal wording → APP 8 accountability exposure and a residual APP 5 governance gap. -- **Evidence:** plain client to `api.openai.com` ([openai.ts:69-73](src/lib/openai.ts)); raw query + - excerpts sent ([rag.ts:7144, 6306](src/lib/rag.ts)); no ZDR/baseURL. -- **Fix (ranked):** (1) Execute an OpenAI DPA and, ideally, obtain **ZDR** for the org; record it in - `docs/`. (2) Obtain governance/legal approval for the shipped draft APP-5 collection/provider +- **Evidence:** the live app and worker are recorded in Railway Singapore + ([deployment-architecture.md](deployment-architecture.md)); the OpenAI client uses + `api.openai.com` ([openai.ts](src/lib/openai.ts)); raw query + excerpts are sent by the RAG pipeline. +- **Fix (ranked):** (1) Record Railway's processor/contractual basis and obtain the applicable APP 8 + determination; execute an OpenAI DPA and, ideally, obtain **ZDR** for the org. (2) Obtain + governance/legal approval for the shipped draft APP-5 collection/provider disclosure and final privacy policy. (3) Retain the shipped on-query/upload PHI reminder. (4) Optionally, add a lightweight PHI-scrub / entity-strip on the outbound query as defence-in-depth. - **Progress (2026-07-13):** fixes (2)+(3) are **live on `main`** via PR #513 @@ -382,25 +402,28 @@ and PHI-minimisation gaps. [scripts/promote-query-misses.ts](scripts/promote-query-misses.ts)), so persistence-off does not affect eval — confirming the pipeline has no real dependency on stored answer text. The flag is additionally blocked in a production-like environment by `npm run check:production-readiness`. -- **Residual (scoped out):** The answer also lands in `rag_response_cache.payload` +- **Residual cache copy:** The answer also lands in `rag_response_cache.payload` ([rag-cache.ts](src/lib/rag-cache.ts)). Its `expires_at` TTL (`RAG_ANSWER_CACHE_TTL_MS`, default 5 min) only gates **reads** — `sharedCacheSelector` filters on `expires_at`, while - `replaceSharedCacheRow` deletes only the _same_ cache key before inserting, and there is **no hard - purge cron** (see the retention table in §8). So an expired row is not destroyed: a one-off query's - answer can persist at rest until a same-key overwrite or a manual invalidation. Nulling the payload - would defeat caching, so it is intentionally not gated by this flag; bounding it properly needs a - scheduled purge (a PIA-4-style follow-up). This is distinct from — and not covered by — the 30-day - durable-log fix above. + `replaceSharedCacheRow` deletes only the _same_ cache key before inserting. Migration + `20260713201542_consolidate_rag_response_cache_retention.sql` unschedules the duplicate unbounded + job and keeps one hourly purge capped at 1,000 expired rows per invocation. This keeps + delete transactions bounded while providing hard cleanup when `pg_cron` is available. Production + was verified live on 2026-07-14: bounded job 16 is active and the obsolete unbounded job is absent. - **Historical cleanup:** a migration to null existing `rag_queries.answer` values is prepared but intentionally unexecuted pending deployment approval; this assessment does not claim live cleanup. -### PIA-4 — `rag_query_misses` never purged **(Medium)** +### PIA-4 — Query-miss and response-cache purges active **(Mitigated)** -- **Risk:** Hashed-query rows accumulate indefinitely; retention policy is inconsistent with - `rag_queries`/`rag_retrieval_logs`. -- **Evidence:** live `cron.job` has no miss-table purge; only jobids 11/12 exist. -- **Fix:** Add a `pg_cron` purge for `rag_query_misses` (30–90 days) mirroring - [migration 20260702120000](supabase/migrations/20260702120000_rag_retrieval_logs_retention.sql). +- **Risk:** A secondary environment without the retention migrations or `pg_cron` can still accumulate + hash-redacted misses and expired response-cache payloads. +- **Evidence:** the original 2026-07-06 live check showed only jobids 11/12. The repository now includes + [migration 20260708120000](supabase/migrations/20260708120000_rag_query_misses_retention.sql), which + installs a 90-day purge, and + [migration 20260713201542](supabase/migrations/20260713201542_consolidate_rag_response_cache_retention.sql), + which installs one bounded hourly response-cache purge when `pg_cron` is available. Production was + queried live on 2026-07-14: jobids 13 and 16 are active and the obsolete duplicate is absent. +- **Fix:** Repeat the canonical job check for each secondary environment that retains real data. ### PIA-5 — Draft notices/page ship; final approved privacy policy remains outstanding **(Medium)** @@ -430,12 +453,13 @@ and PHI-minimisation gaps. ## 11. Recommendation -Before the app is used with real patients in a WA clinical setting, close **PIA-1** (execute the -cross-border DPA/ZDR contractual basis and approve the shipped draft APP 5 wording) and **PIA-2** (place +Before the app is used with real patients in a WA clinical setting, close **PIA-1** (record the Railway +Singapore processor/APP 8 basis, execute the OpenAI DPA/ZDR basis, and approve the shipped draft APP 5 +wording) and **PIA-2** (place the mandatory HMAC secret in the deploy host's secret store; the fail-closed boot guard is now enforced in code) as launch-blockers. **PIA-3** is mitigated (the durable `rag_queries.answer` log is no -longer persisted by default; gated behind `RAG_PERSIST_ANSWER_TEXT`); **PIA-4** remains as a fast -follow-up (purge `rag_query_misses`, and add a `rag_response_cache` purge cron), and complete the +longer persisted by default; gated behind `RAG_PERSIST_ANSWER_TEXT`); **PIA-4** is mitigated by the +committed query-miss and bounded response-cache purges, verified live on 2026-07-14. Complete the **PIA-5** residual data-handling documentation. The data-at-rest security posture (Sydney residency, RLS, private storage, query hashing, automated purge) is already strong and should be highlighted in the privacy policy as evidence of "reasonable steps" under APP 11. diff --git a/docs/rag-hybrid-findings-and-todo.md b/docs/rag-hybrid-findings-and-todo.md index 4c65954f31..01f1afdeb6 100644 --- a/docs/rag-hybrid-findings-and-todo.md +++ b/docs/rag-hybrid-findings-and-todo.md @@ -1,8 +1,9 @@ # RAG Hybrid Retrieval — Findings & To-Do (2026-07-01) Living list of issues found while fixing the live-only hybrid-RPC schema drift and optimising the -online RAG. Grouped by priority. Anything marked ✅ is done + validated this workstream; ⏳ is the -outstanding backlog. See also the master plan +online RAG. Grouped by priority. **Last reconciled: 2026-07-14.** `✅` is complete/closed, `🔶` is +partially complete with an explicit remaining action, and `⏳` is genuinely open. Historical +measurements are evidence, not instructions to repeat provider-backed work without approval. See also the master plan (`C:\Users\joshs\.claude\plans\please-review-the-current-synthetic-pinwheel.md`) for RC IDs and `docs/search-rag-master-plan.md`. @@ -113,10 +114,9 @@ denied to set parameter`)** — the RC11 blocker. The only method hosted allows ## P2 — latency, eval coverage, data -7. ⏳ **p90 retrieval ~8.6s on hybrid cases.** Multiple sequential Supabase RPC round-trips per query - (embedding + chunks + table_facts + embedding_fields + index_units + memory_cards + rerank). Some of - this is local-machine→remote-DB network latency (prod is co-located), but consider firing the - independent layer RPCs in parallel and/or trimming layers that don't move recall. +7. 🔶 **Historical p90 retrieval ~8.6s; superseded by item 25.** Multiple sequential Supabase RPC + round-trips remain relevant, but the approved 2026-07-14 live comparison found a much wider tail + driven by slow hybrid RPC execution. Use item 25 for the current evidence and next action. 8. ✅ **Golden eval set expanded 10 → 23 (2026-07-01).** Added 12 verified cases built from real corpus content (condition guidelines — bipolar, alcohol, opioid, schizophrenia, insomnia, suicide, depression — which the original EMHS-only set lacked) across broad_summary/comparison/ @@ -231,11 +231,19 @@ denied to set parameter`)** — the RC11 blocker. The only method hosted allows ## P2 — offline/fallback remainder (Workstream F) -11. ⏳ Global **AI-status indicator** + health probe (is OpenAI reachable/degraded). -12. ⏳ **Answer cache for true offline reuse** (`rag_response_cache` `cache_kind='answer'`), marked "cached". -13. ⏳ Tag the **auto-degrade generation-failure** fallback (`buildGenerationFallbackAnswer` returns - before the labelling wrapper, so it isn't stamped `source_only`). -14. ⏳ Playwright assertion for the `source-only-disclosure` badge (needs the running app). +11. 🔶 **AI degradation visibility — PARTIAL.** Deep health exposes configured provider mode plus + trailing degraded-answer and failure counters without making a paid provider call. Remaining: + wire the existing thresholds/nightly canary into an operator alert channel. Do not add an active + OpenAI request to the ordinary health endpoint merely to report reachability. +12. ✅ **Answer cache for offline/repeat reuse — DONE.** Process-local and owner-scoped shared + `rag_response_cache` `cache_kind='answer'` paths are implemented and labelled as cached. The + 2026-07-14 retention reconciliation removes the duplicate unbounded purge job and keeps one + hourly purge capped at 1,000 expired rows. Applied live 2026-07-14: bounded job 16 is active, the + duplicate unbounded job is absent, and the cache contained no expired rows at verification time. +13. ✅ **Auto-degrade generation fallback labelling — DONE.** Final answer quality labelling stamps + source-only answers with `answerQualityTier`, `fallbackReason`, and explicit `degradedMode`. +14. ✅ **Source-only disclosure browser assertion — DONE.** `tests/ui-smoke.spec.ts` asserts the + `source-only-disclosure` badge and verify-against-sources guidance. ## P2 — naturalness residual @@ -325,26 +333,35 @@ denied to set parameter`)** — the RC11 blocker. The only method hosted allows registry `POST`/`PATCH`/`PUT` path must call `bestEffortReembedRegistryRecordAfterEdit` after the write commits. Remaining non-blocking UX follow-up: an answer-mode check that registry-backed citations render as curated registry records rather than primary source documents. -23. ⏳ **Finding #11 full fix (RAG optimisation Phase 2)** — the classifier-verdict memo (shipped - 2026-07-06) makes zero-result behaviour deterministic per query but does not close the gap: - the deterministic analyzer still cannot tell in-corpus topics from out-of-corpus ones. - Phase-2 spec stands (corpus-grounded relevance: IDF/corpus-frequency weighting of query - terms + data-driven vocabulary), with the added prerequisite that item 17's vocabulary - capture now supplies real miss data to seed the vocabulary from. +23. ✅ **Finding #11 full fix — CLOSED.** The corpus-grounded relevance implementation described in + item 10 (2026-07-07) superseded the earlier classifier-memo-only state. In-corpus bare topics and + corpus-absent invented terms now follow deterministic corpus evidence; no duplicate Phase-2 task + remains. Item 17's broader alias-promotion privacy/design work remains separate. 24. ⏳ **OCR dropped-letter corruption in table index units** — no reliable detector exists (82% false positives; guard reverted). Next viable angle: dictionary-based repair at INGESTION (compare table-cell tokens against the document's own clean chunk text — "p ycho ocial" aligns to "psychosocial" within the same page's raw text) rather than heuristic detection at query time. Scope to `worker/` table extraction; requires the Python OCR stack to test. -25. ⏳ **Retrieval/RAG latency remains a performance backlog, not a current gate blocker.** +25. 🔶 **Retrieval/RAG latency — serial-depth fixes landed locally; live tail isolated to DB RPCs.** Post-registry retrieval stayed quality-clean (`top_k_hit_rate=1`, `document_recall_at_5=1`, `content_recall_at_5=1`, `force_embedding_failure_count=0`) with local `p90_latency_ms=13145`. Post-routing RAG-only passed with `p95_latency_ms=20385` and no blocking threshold failures, but - local-machine→remote-DB latency and remaining sequential layers still justify a dedicated perf pass. - Next step (measure first): instrument the existing per-request `documentRankingMetadataCache` to - quantify incremental cache misses/fetches before changing enrichment, then test starting - `embedTextWithTelemetry` concurrently only when `forceEmbedding` is set or routing has already ruled - out an accepted lexical/document fast path. Do not preload embeddings unconditionally: preserve - source-only and lexical-only behavior, and record provider-call count plus whether each embedding - result was consumed so any p90 gain is weighed against cost. Gate with the golden eval unchanged, - provider usage unchanged or explicitly accepted, and p90 from `rag_retrieval_logs` before/after. + local-machine→remote-DB latency and remaining sequential layers still justify a measured perf pass. + The 2026-07-14 local fix removes the second blocking `rag_response_cache` miss-diagnostic query and + starts document metadata/index-quality plus page/direct-image hydration reads concurrently. Focused + tests prove one shared-cache read per filtered miss and both concurrency boundaries. + + **Approved live comparison (2026-07-14):** a warmed repeat reported `supabase_rpc_latency_ms=0`, + while two cold synthesis probes spent 48.5–49.4s in Supabase RPCs and 51–53s total. A six-case + live-database retrieval run on the local code kept `document_recall_at_5=1`, + `content_recall_at_5=1`, and `top_k_hit_rate=1`, with 1.8s median / 47.3s p90. Database statement + statistics identify the legacy hybrid chunk/embedding-field RPC families as the tail source, with + material temporary I/O. Forced-embedding/model-routing evaluation stopped after the provider + returned quota exhaustion; no model or production configuration was changed. + + **Next smallest performance work:** capture `EXPLAIN (ANALYZE, BUFFERS)` for the slow RPC shapes + with non-sensitive fixtures, optimise those plans, and then compare Micro versus Small primary + compute if the plans remain memory-bound. Do not create a Singapore read replica yet: replicas + require at least Small compute and would duplicate the same slow plans while adding asynchronous + freshness/read-routing work. Reconsider only after database execution is fast enough that network + RTT is again a material share. diff --git a/src/app/privacy/page.tsx b/src/app/privacy/page.tsx index c843595366..a5de27ca2d 100644 --- a/src/app/privacy/page.tsx +++ b/src/app/privacy/page.tsx @@ -46,22 +46,23 @@ const SECTIONS: Section[] = [ ), }, { - heading: "Where data is stored", - body: "Documents, extracted evidence, metadata, account records, and owner-scoped operational records are stored in the configured Supabase project. File buckets are private and links are time-limited. The operator must verify the deployed project region and contractual controls.", + heading: "Where data is stored and processed", + body: "Documents, extracted evidence, metadata, account records, and owner-scoped operational records are stored in the configured Supabase project in Sydney. The production application and ingestion worker currently run on Railway in Singapore, so questions, retrieved evidence, answers, and ingestion material are processed in or transit through Singapore. File buckets are private and links are time-limited. The operator must verify deployed regions and contractual controls.", }, { heading: "External provider processing", body: ( <> - When model-backed answering is enabled, the question and selected source excerpts are sent to the configured - OpenAI API. This processing may occur outside Australia. Provider mode can also return a local source-only - response. The operator must verify provider regions, retention terms, contracts, and cross-border obligations. + When external provider mode is configured, question text may be sent to the OpenAI API to create a retrieval + embedding, including when the final response is source-only. When model-backed answer synthesis is used, the + question and selected source excerpts are also sent. This processing may occur outside Australia. The operator + must verify provider regions, retention terms, contracts, and cross-border obligations. ), }, { heading: "Retention", - body: "Repository migrations configure 30-day retention for RAG query records and 90-day retention for retrieval logs when the database scheduler is available. Query-miss and expired response-cache cleanup require separate governance and operational controls. Uploaded documents remain until removed under the applicable process.", + body: "Repository migrations configure 30-day retention for RAG query records, 90-day retention for retrieval logs and query-miss telemetry, and a bounded hourly purge of expired response-cache rows when the database scheduler is available. The operator must verify that those scheduled jobs are active. Uploaded documents remain until removed under the applicable process.", }, { heading: "Your responsibilities", @@ -91,8 +92,10 @@ export default function PrivacyPage() {

- Do not enter identifiable patient details such as names, dates of birth, or record numbers. When - model-backed answering is enabled, your question is sent to the configured OpenAI API. + Do not enter identifiable patient details such as names, dates of birth, or record numbers. Requests are + processed by the application service in Singapore. With external provider mode configured, question text + may be sent to the OpenAI API for retrieval embedding even when the final response is source-only; + model-backed answer synthesis also sends the question and selected evidence.

diff --git a/src/lib/health-response.ts b/src/lib/health-response.ts index 85c8a5a8c2..29df558351 100644 --- a/src/lib/health-response.ts +++ b/src/lib/health-response.ts @@ -1,7 +1,7 @@ import { NextResponse } from "next/server"; import { allowDeepHealthProbe } from "@/lib/deep-probe-auth"; import { env, isDemoMode } from "@/lib/env"; -import type { AnswerSloSnapshot } from "@/lib/observability/answer-slo"; +import type { AnswerSloSnapshot, SloProbeClient } from "@/lib/observability/answer-slo"; import { cacheMetricsSnapshot, type CacheMetricsSnapshot } from "@/lib/observability/cache-metrics"; import type { SpendProbeClient, SpendSnapshot } from "@/lib/observability/spend-metrics"; @@ -51,7 +51,9 @@ export async function healthResponse(request: Request, options: HealthResponseOp checks.supabase = health.ok ? "ok" : "error"; if (health.ok && options.includeSlo !== false) { try { - slo = await answerSloSnapshot(admin); + // Avoid recursively instantiating the full generated PostgREST + // client type against the intentionally tiny SLO query surface. + slo = await answerSloSnapshot(admin as unknown as SloProbeClient); } catch { slo = null; } diff --git a/src/lib/observability/answer-slo.ts b/src/lib/observability/answer-slo.ts index a4b69c2860..e27148022e 100644 --- a/src/lib/observability/answer-slo.ts +++ b/src/lib/observability/answer-slo.ts @@ -7,9 +7,8 @@ // into a scrapeable counter so host-native alerting can poll them instead of // running the SQL by hand. It only aggregates `rag_queries.metadata` (never raw // query text, which is redacted at write time), and runs behind the secret-gated -// deep probe. The cache hit-rate counter from that doc is intentionally NOT here — -// it needs in-process instrumentation of the retrieval hot path (rag-cache.ts) and -// is tracked as a separate follow-up. +// deep probe. Cache hit-rate is exposed separately by cache-metrics.ts because it +// is an in-process hot-path counter rather than a trailing database aggregate. export type AnswerSloSnapshot = { windowMinutes: number; @@ -37,9 +36,11 @@ type CountResult = { count: number | null; error: unknown }; // route passes it directly and tests pass a small fake. type SloCountBuilder = PromiseLike & { gt(column: string, value: string): SloCountBuilder; + eq(column: string, value: string): SloCountBuilder; is(column: string, value: null): SloCountBuilder; not(column: string, operator: string, value: null): SloCountBuilder; ilike(column: string, pattern: string): SloCountBuilder; + or(filters: string): SloCountBuilder; }; export type SloProbeClient = { @@ -54,7 +55,7 @@ function rate(numerator: number, denominator: number) { /** * Count answered queries in the trailing window and how many carried a - * `hybrid_rpc_errors` map or a `fallback_reason` (degraded/source-only). Throws on + * `hybrid_rpc_errors` map or an explicit provider-generation fallback flag. Throws on * a query error so the caller can mark the probe degraded rather than report a * falsely-healthy zero. */ @@ -73,7 +74,12 @@ export async function answerSloSnapshot(client: SloProbeClient, windowMinutes = const [total, hybrid, degraded, truncation, timeout] = await Promise.all([ base(), base().not("metadata->hybrid_rpc_errors", "is", null), - base().not("metadata->>fallback_reason", "is", null), + // Source-only/extractive answers can be intentional and healthy. Count only + // model-generation failures that actually forced a local fallback. Keep the + // legacy reason match until pre-flag rows have aged out of every SLO window. + base().or( + "metadata->>provider_generation_degraded.eq.true,metadata->>fallback_reason.ilike.%generation_fallback:%", + ), // fallback_reason values look like "...generation_fallback:provider_incomplete_max_output_tokens" // and "...generation_fallback:provider_timeout" (confirmed in live rag_queries). base().ilike("metadata->>fallback_reason", "%max_output_tokens%"), diff --git a/src/lib/rag-answer-support.ts b/src/lib/rag-answer-support.ts index 31416193a1..f91c33f2a0 100644 --- a/src/lib/rag-answer-support.ts +++ b/src/lib/rag-answer-support.ts @@ -52,6 +52,11 @@ export function fallbackReasonFromRouting(reason?: string | null) { ); } +/** True only when model generation failed and the answer fell back locally. */ +export function isProviderGenerationDegraded(reason?: string | null) { + return /(?:^|;\s*)generation_fallback(?::|$)/i.test(reason ?? ""); +} + export function collectMemoryCards(results: SearchResult[], limit = 8) { const seen = new Set(); const cards: DocumentMemoryCard[] = []; diff --git a/src/lib/rag-cache.ts b/src/lib/rag-cache.ts index afd3b730e3..51bb512b3e 100644 --- a/src/lib/rag-cache.ts +++ b/src/lib/rag-cache.ts @@ -268,14 +268,7 @@ export async function setCachedSearch( type SharedCacheKind = "search" | "answer"; type SharedCacheMissReason = - | "cache_lookup_error" - | "cache_lookup_exception" - | "cache_payload_invalid" - | "no_entry" - | "expired" - | "indexing_version_mismatch" - | "dependency_version_mismatch" - | "unknown_filter_miss"; + "cache_lookup_error" | "cache_lookup_exception" | "cache_payload_invalid" | "unknown_filter_miss"; function sharedCacheSelector( supabase: ReturnType, @@ -367,37 +360,6 @@ export async function getSharedCachedSearch( if (args.skipCache || env.RAG_SEARCH_CACHE_TTL_MS <= 0) return null; const normalizedQuery = retrievalPlanCacheQuery(args, queryClass, queryVariants); const indexingVersion = await cacheIndexingVersion(args); - async function probeSharedCacheMissReason(reasonFromLookup?: SharedCacheMissReason): Promise { - if (reasonFromLookup) return reasonFromLookup; - try { - const supabase = createAdminClient(); - let probeQuery = supabase - .from("rag_response_cache") - .select("indexing_version,dependency_version,expires_at") - .eq("cache_kind", "search") - .eq("scope_key", scopeKey(args)) - .eq("normalized_query", normalizedQuery) - .order("expires_at", { ascending: false }) - .limit(5); - probeQuery = args.ownerId ? probeQuery.eq("owner_id", args.ownerId) : probeQuery.is("owner_id", null); - const { data, error } = await probeQuery; - if (error) return "cache_lookup_error"; - if (!data?.length) return "no_entry"; - const now = Date.now(); - const nonExpired = data.find((entry) => { - const expiresAt = Date.parse(String(entry.expires_at ?? "")); - return Number.isFinite(expiresAt) && expiresAt > now; - }); - if (!nonExpired) return "expired"; - if (String(nonExpired.indexing_version ?? "") !== indexingVersion) return "indexing_version_mismatch"; - if (String(nonExpired.dependency_version ?? "") !== ragCacheDependencyVersion) { - return "dependency_version_mismatch"; - } - return "unknown_filter_miss"; - } catch { - return "cache_lookup_exception"; - } - } try { const { data, error } = await sharedCacheSelector( createAdminClient(), @@ -406,11 +368,14 @@ export async function getSharedCachedSearch( indexingVersion, normalizedQuery, ).maybeSingle(); - if (error) return { kind: "miss", reason: await probeSharedCacheMissReason("cache_lookup_error") }; - if (!data?.payload) return { kind: "miss", reason: await probeSharedCacheMissReason() }; + if (error) return { kind: "miss", reason: "cache_lookup_error" }; + // The selector deliberately folds TTL, dependency, and indexing-version + // validity into the hit query. Keep a filtered miss coarse rather than pay + // a second cross-region lookup solely to distinguish those miss classes. + if (!data?.payload) return { kind: "miss", reason: "unknown_filter_miss" }; const payload = data.payload as { results?: SearchResult[]; telemetry?: Partial }; if (!Array.isArray(payload.results)) { - return { kind: "miss", reason: await probeSharedCacheMissReason("cache_payload_invalid") }; + return { kind: "miss", reason: "cache_payload_invalid" }; } return { kind: "hit", diff --git a/src/lib/rag.ts b/src/lib/rag.ts index 0d77945873..f6c6d0ce4a 100644 --- a/src/lib/rag.ts +++ b/src/lib/rag.ts @@ -152,6 +152,7 @@ import { deriveConfidence, evidenceTextForGate, fallbackReasonFromRouting, + isProviderGenerationDegraded, machineReadableFallbackAnswer, scoreValue, } from "@/lib/rag-answer-support"; @@ -162,6 +163,7 @@ export { deriveConfidence, evidenceTextForGate, fallbackReasonFromRouting, + isProviderGenerationDegraded, machineReadableFallbackAnswer, rankMemoryCardsForAnswer, scoreValue, @@ -2526,7 +2528,7 @@ function createDocumentRankingMetadataCache(): DocumentRankingMetadataCache { } /** Attach document ranking metadata. */ -async function attachDocumentRankingMetadata( +export async function attachDocumentRankingMetadata( supabase: ReturnType, results: SearchResult[], ownerId?: string, @@ -2563,17 +2565,22 @@ async function attachDocumentRankingMetadata( return attachIndexQualityMetadata(supabase, enriched, ownerId, cache); } - try { - const metadataRows = await fetchRelatedDocumentMetadata({ + const [metadataRows, indexedResults] = await Promise.all([ + fetchRelatedDocumentMetadata({ supabase, ownerId, documentIds: missingDocumentIds, - }); + }).catch(() => null), + attachIndexQualityMetadata(supabase, results, ownerId, cache), + ]); + if (!metadataRows) return indexedResults; + + try { for (const documentId of missingDocumentIds) cache.documentMetadata.set(documentId, null); for (const row of metadataRows) { cache.documentMetadata.set(row.document_id, { labels: row.labels, summary: row.summary }); } - const enriched = results.map((result) => { + return indexedResults.map((result) => { const metadata = cache.documentMetadata.get(result.document_id); if (!metadata) return result; return { @@ -2582,9 +2589,8 @@ async function attachDocumentRankingMetadata( document_summary: metadata.summary, }; }); - return attachIndexQualityMetadata(supabase, enriched, ownerId, cache); } catch { - return results; + return indexedResults; } } @@ -2624,7 +2630,7 @@ async function attachIndexQualityMetadata( } /** Attach page visual evidence. */ -async function attachPageVisualEvidence( +export async function attachPageVisualEvidence( supabase: ReturnType, results: SearchResult[], ): Promise { @@ -2646,9 +2652,9 @@ async function attachPageVisualEvidence( const selectColumns = "id,document_id,page_number,storage_path,caption,bbox,image_type,searchable,clinical_relevance_score,source_kind,width,height,labels,metadata"; - const pageData = + const [pageData, directData] = await Promise.all([ pageNumbers.length > 0 - ? await supabase + ? supabase .from("document_images") .select(selectColumns) .in("document_id", documentIds) @@ -2657,17 +2663,17 @@ async function attachPageVisualEvidence( .neq("image_type", "logo_decorative") .order("clinical_relevance_score", { ascending: false }) .limit(80) - : { data: [], error: null }; - const directData = + : Promise.resolve({ data: [], error: null }), sourceImageIds.length > 0 - ? await supabase + ? supabase .from("document_images") .select(selectColumns) .in("id", sourceImageIds) .eq("searchable", true) .neq("image_type", "logo_decorative") .limit(sourceImageIds.length) - : { data: [], error: null }; + : Promise.resolve({ data: [], error: null }), + ]); const data = [...(pageData.data ?? []), ...(directData.data ?? [])]; if ((pageData.error && directData.error) || data.length === 0) return results; @@ -4548,6 +4554,8 @@ async function answerQuestionWithScopeUncoalesced( routing_reason: route.reason, query_class: queryClass, fallback_reason: fallbackReasonFromRouting(route.reason), + degraded: finalizedAnswer.degradedMode?.active ?? false, + provider_generation_degraded: isProviderGenerationDegraded(finalizedAnswer.routingReason), model_used: null, retrieved_candidate_count: results.length, ...smartApiLogMetadata(smartApiPlan), @@ -4676,6 +4684,8 @@ async function answerQuestionWithScopeUncoalesced( routing_reason: finalizedAnswer.routingReason, query_class: queryClass, fallback_reason: fallbackReasonFromRouting(finalizedAnswer.routingReason), + degraded: finalizedAnswer.degradedMode?.active ?? false, + provider_generation_degraded: isProviderGenerationDegraded(finalizedAnswer.routingReason), model_used: null, retrieved_candidate_count: results.length, ...smartApiLogMetadata(smartApiPlan), @@ -5380,6 +5390,8 @@ ${qualityRetryInstruction}` routing_reason: routingReason, query_class: queryClass, fallback_reason: fallbackReasonFromRouting(answer.routingReason), + degraded: answer.degradedMode?.active ?? false, + provider_generation_degraded: isProviderGenerationDegraded(answer.routingReason), model_used: modelUsed, requested_fast_model: requestedOpenAIAnswerModels.fastAnswer, requested_strong_model: requestedOpenAIAnswerModels.strongAnswer, @@ -5630,6 +5642,8 @@ ${qualityRetryInstruction}` routing_reason: fallbackAnswer.routingReason, query_class: queryClass, fallback_reason: fallbackReasonFromRouting(fallbackAnswer.routingReason), + degraded: fallbackAnswer.degradedMode?.active ?? false, + provider_generation_degraded: isProviderGenerationDegraded(fallbackAnswer.routingReason), model_used: null, requested_fast_model: requestedOpenAIAnswerModels.fastAnswer, requested_strong_model: requestedOpenAIAnswerModels.strongAnswer, diff --git a/supabase/migrations/20260713201542_consolidate_rag_response_cache_retention.sql b/supabase/migrations/20260713201542_consolidate_rag_response_cache_retention.sql new file mode 100644 index 0000000000..2e38bd4d6d --- /dev/null +++ b/supabase/migrations/20260713201542_consolidate_rag_response_cache_retention.sql @@ -0,0 +1,22 @@ +-- Two earlier migrations installed separate response-cache purge jobs. Keep +-- only the job that calls the bounded implementation so cleanup cannot turn a +-- large cache backlog into a single long-running delete. +set search_path = public, pg_catalog, pg_temp; + +do $cron$ +begin + if to_regnamespace('cron') is null then + return; + end if; + + perform cron.unschedule(j.jobid) + from cron.job j + where j.jobname in ('purge-rag-response-cache', 'purge-expired-rag-response-cache'); + + perform cron.schedule( + 'purge-rag-response-cache', + '15 * * * *', + $job$select public.purge_expired_rag_response_cache(1000);$job$ + ); +end +$cron$; diff --git a/tests/answer-slo.test.ts b/tests/answer-slo.test.ts index 59cb766f74..28855e8d67 100644 --- a/tests/answer-slo.test.ts +++ b/tests/answer-slo.test.ts @@ -3,9 +3,9 @@ import { describe, expect, it } from "vitest"; import { answerSloSnapshot, type SloProbeClient } from "@/lib/observability/answer-slo"; // Fake PostgREST count builder: from().select().gt() is the "total" query; adding -// .not(column,...) narrows it to the hybrid-error or degraded count, and .ilike(col, -// pattern) narrows it to the truncation or timeout fallback subset by pattern; the third -// arg records the base .is(...) filters so tests can assert event-type scoping. Awaiting +// .not(column,...) narrows it to the hybrid-error count, .or(filters) narrows it +// to provider-generation fallbacks, and .ilike(col, pattern) narrows it to the +// truncation or timeout fallback subset by pattern. Awaiting // resolves to { count, error }. type SloFilterKey = "total" | "hybrid" | "degraded" | "truncation" | "timeout"; @@ -13,6 +13,7 @@ function fakeClient( counts: { total: number; hybrid: number; degraded: number; truncation?: number; timeout?: number }, error?: unknown, observedBaseFilters: Array<{ column: string; value: null }> = [], + observedNarrowingFilters: Array<{ method: "eq" | "not" | "or"; column: string; value: unknown }> = [], ): SloProbeClient { const build = (filter: SloFilterKey) => { const builder = { @@ -21,7 +22,18 @@ function fakeClient( observedBaseFilters.push({ column, value }); return builder; }, - not: (column: string) => build(column.includes("hybrid_rpc_errors") ? "hybrid" : "degraded"), + eq: (column: string, value: string) => { + observedNarrowingFilters.push({ method: "eq", column, value }); + return build("degraded"); + }, + not: (column: string, _operator: string, value: null) => { + observedNarrowingFilters.push({ method: "not", column, value }); + return build("hybrid"); + }, + or: (filters: string) => { + observedNarrowingFilters.push({ method: "or", column: "", value: filters }); + return build("degraded"); + }, ilike: (_column: string, pattern: string) => build(pattern.includes("max_output_tokens") ? "truncation" : "timeout"), then: (resolve: (value: { count: number | null; error: unknown }) => unknown) => @@ -65,6 +77,25 @@ describe("answerSloSnapshot", () => { ); }); + it("counts only provider-generation fallbacks, not intentional source-only answers", async () => { + const observedNarrowingFilters: Array<{ + method: "eq" | "not" | "or"; + column: string; + value: unknown; + }> = []; + + await answerSloSnapshot(fakeClient({ total: 7, hybrid: 1, degraded: 2 }, undefined, [], observedNarrowingFilters)); + + expect(observedNarrowingFilters).toContainEqual({ + method: "or", + column: "", + value: "metadata->>provider_generation_degraded.eq.true,metadata->>fallback_reason.ilike.%generation_fallback:%", + }); + expect(observedNarrowingFilters).not.toContainEqual( + expect.objectContaining({ method: "not", column: "metadata->>fallback_reason" }), + ); + }); + it("reports zero rates (not NaN) when there are no queries in the window", async () => { const snapshot = await answerSloSnapshot(fakeClient({ total: 0, hybrid: 0, degraded: 0 })); expect(snapshot.totalQueries).toBe(0); diff --git a/tests/privacy-ui.test.ts b/tests/privacy-ui.test.ts index ee12c1dd6f..1192f575dd 100644 --- a/tests/privacy-ui.test.ts +++ b/tests/privacy-ui.test.ts @@ -25,7 +25,7 @@ describe("privacy UI", () => { "What this tool is", "What is collected", "How questions are handled", - "Where data is stored", + "Where data is stored and processed", "External provider processing", "Retention", "Your responsibilities", @@ -33,6 +33,11 @@ describe("privacy UI", () => { expect(markup).toContain(heading); } expect(markup).toContain("Generated answer text is also omitted from durable query logs by default"); + expect(markup).toContain("application service in Singapore"); + expect(markup).toContain("Railway in Singapore"); + expect(markup).toContain("retrieval embedding"); + expect(markup).toContain("even when the final response is source-only"); + expect(markup).toContain("bounded hourly purge of expired response-cache rows"); expect(markup).not.toContain("approved privacy policy"); }); }); diff --git a/tests/rag-answer-fallback.test.ts b/tests/rag-answer-fallback.test.ts index 57459a4398..4e5354eb53 100644 --- a/tests/rag-answer-fallback.test.ts +++ b/tests/rag-answer-fallback.test.ts @@ -992,6 +992,8 @@ describe("RAG structured-output fallback", () => { const loggedMetadata = loggedRow.metadata ?? {}; expect(loggedMetadata.answer_retry_count).toBe(2); expect(loggedMetadata.answer_retry_reasons).toEqual(["fast_template_retry_strong", "strong_quality_retry"]); + expect(loggedMetadata.degraded).toBe(false); + expect(loggedMetadata.provider_generation_degraded).toBe(false); // PIA-3: the generated answer text must not be persisted to rag_queries.answer // unless RAG_PERSIST_ANSWER_TEXT is enabled (default off), and the row records // that the answer was not retained. diff --git a/tests/rag-answer-support.test.ts b/tests/rag-answer-support.test.ts new file mode 100644 index 0000000000..b4d3a2a9c0 --- /dev/null +++ b/tests/rag-answer-support.test.ts @@ -0,0 +1,12 @@ +import { describe, expect, it } from "vitest"; + +import { isProviderGenerationDegraded } from "@/lib/rag-answer-support"; + +describe("provider generation degradation classification", () => { + it("counts generation fallbacks but excludes intentional extractive and unsupported routes", () => { + expect(isProviderGenerationDegraded("strong_generation; generation_fallback:provider_timeout")).toBe(true); + expect(isProviderGenerationDegraded("high_confidence_extractive_retrieval")).toBe(false); + expect(isProviderGenerationDegraded("source_support_document_lookup")).toBe(false); + expect(isProviderGenerationDegraded("confidence_gate_blocked; unsupported")).toBe(false); + }); +}); diff --git a/tests/rag-query-concurrency.test.ts b/tests/rag-query-concurrency.test.ts new file mode 100644 index 0000000000..0f93d93f89 --- /dev/null +++ b/tests/rag-query-concurrency.test.ts @@ -0,0 +1,136 @@ +import { afterEach, describe, expect, it, vi } from "vitest"; + +import type { SearchResult } from "../src/lib/types"; + +function deferred() { + let resolve!: (value: T) => void; + const promise = new Promise((promiseResolve) => { + resolve = promiseResolve; + }); + return { promise, resolve }; +} + +function result(indexUnit?: SearchResult["index_unit"]): SearchResult { + return { + id: "chunk-1", + document_id: "document-1", + title: "Lithium guideline", + file_name: "lithium.pdf", + page_number: 1, + chunk_index: 0, + section_heading: null, + content: "Lithium monitoring guidance.", + image_ids: [], + similarity: 0.9, + text_rank: 0.8, + hybrid_score: 0.9, + document_labels: [], + images: [], + index_unit: indexUnit, + }; +} + +afterEach(() => { + vi.restoreAllMocks(); + vi.resetModules(); +}); + +describe("RAG enrichment query concurrency", () => { + it("starts document metadata and index-quality reads together", async () => { + vi.resetModules(); + const metadata = deferred>(); + const quality = deferred<{ data: unknown[]; error: null }>(); + const started: string[] = []; + + vi.doMock("@/lib/document-enrichment", async () => { + const actual = + await vi.importActual("@/lib/document-enrichment"); + return { + ...actual, + fetchRelatedDocumentMetadata: vi.fn(() => { + started.push("metadata"); + return metadata.promise; + }), + }; + }); + + const query = { + select: () => query, + in: () => query, + eq: () => query, + then: (resolve: (value: { data: unknown[]; error: null }) => unknown) => { + started.push("quality"); + return quality.promise.then(resolve); + }, + }; + const supabase = { from: vi.fn(() => query) }; + const { attachDocumentRankingMetadata } = await import("../src/lib/rag"); + + const pending = attachDocumentRankingMetadata( + supabase as never, + [result()], + "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa", + ); + await vi.waitFor(() => expect(started).toEqual(expect.arrayContaining(["metadata", "quality"]))); + + metadata.resolve([{ document_id: "document-1", labels: ["lithium"], summary: "Monitoring summary" }]); + quality.resolve({ data: [], error: null }); + + await expect(pending).resolves.toEqual([ + expect.objectContaining({ document_labels: ["lithium"], document_summary: "Monitoring summary" }), + ]); + }); + + it("starts page and direct-image reads together", async () => { + vi.resetModules(); + const page = deferred<{ data: unknown[]; error: null }>(); + const direct = deferred<{ data: unknown[]; error: null }>(); + const started: string[] = []; + + const supabase = { + from: vi.fn(() => { + let kind = "page"; + const query = { + select: () => query, + in: (column: string) => { + if (column === "id") kind = "direct"; + return query; + }, + eq: () => query, + neq: () => query, + order: () => query, + limit: () => query, + then: (resolve: (value: { data: unknown[]; error: null }) => unknown) => { + started.push(kind); + return (kind === "direct" ? direct.promise : page.promise).then(resolve); + }, + }; + return query; + }), + }; + const { attachPageVisualEvidence } = await import("../src/lib/rag"); + const searchResult = result({ + id: "unit-1", + unit_type: "table", + title: "Lithium table", + content: "Lithium table", + source_chunk_id: "chunk-1", + source_image_id: "image-1", + page_start: 1, + page_end: 1, + heading_path: [], + normalized_terms: ["lithium"], + quality_score: 1, + extraction_mode: "deterministic", + metadata: {}, + }); + + const pending = attachPageVisualEvidence(supabase as never, [searchResult]); + await vi.waitFor(() => expect(started).toEqual(expect.arrayContaining(["page", "direct"]))); + + page.resolve({ data: [], error: null }); + direct.resolve({ data: [], error: null }); + + await expect(pending).resolves.toEqual([searchResult]); + }); +}); diff --git a/tests/rag-shared-cache.test.ts b/tests/rag-shared-cache.test.ts new file mode 100644 index 0000000000..614cdae28d --- /dev/null +++ b/tests/rag-shared-cache.test.ts @@ -0,0 +1,65 @@ +import { afterEach, describe, expect, it, vi } from "vitest"; + +afterEach(() => { + vi.restoreAllMocks(); + vi.resetModules(); +}); + +describe("shared RAG search cache", () => { + it("uses one shared-cache read for a cold filtered miss", async () => { + vi.resetModules(); + let sharedCacheReads = 0; + + vi.doMock("@/lib/env", () => ({ + env: { + RAG_SEARCH_CACHE_TTL_MS: 60_000, + RAG_SEARCH_CACHE_SIZE: 200, + RAG_PERSIST_RAW_QUERY_TEXT: false, + RAG_QUERY_HASH_SECRET: "test-query-hash-secret", + }, + isDemoMode: () => false, + isLocalNoAuthMode: () => false, + })); + vi.doMock("@/lib/deep-memory", () => ({ ragDeepMemoryVersion: "test-rag-version" })); + vi.doMock("@/lib/clinical-search", () => ({ + buildClinicalTextSearchQuery: (query: string) => query.trim(), + })); + vi.doMock("@/lib/supabase/admin", () => ({ + createAdminClient: () => ({ + from: (table: string) => { + const builder = { + select: () => builder, + eq: () => builder, + is: () => builder, + in: () => builder, + or: () => builder, + gt: () => builder, + order: () => builder, + limit: () => builder, + maybeSingle: async () => { + expect(table).toBe("rag_response_cache"); + sharedCacheReads += 1; + return { data: null, error: null }; + }, + then: (resolve: (value: { data: unknown[]; error: null }) => unknown) => + Promise.resolve({ + data: + table === "documents" ? [{ id: "doc-1", updated_at: "2026-07-14T00:00:00.000Z", metadata: {} }] : [], + error: null, + }).then(resolve), + }; + return builder; + }, + }), + })); + + const { getSharedCachedSearch } = await import("../src/lib/rag-cache"); + const result = await getSharedCachedSearch({ + query: "lithium monitoring", + ownerId: "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa", + }); + + expect(result).toEqual({ kind: "miss", reason: "unknown_filter_miss" }); + expect(sharedCacheReads).toBe(1); + }); +}); diff --git a/tests/supabase-schema.test.ts b/tests/supabase-schema.test.ts index abf191e241..ed28583914 100644 --- a/tests/supabase-schema.test.ts +++ b/tests/supabase-schema.test.ts @@ -164,6 +164,10 @@ const ragQueryMissesRetentionMigration = readFileSync( new URL("../supabase/migrations/20260708120000_rag_query_misses_retention.sql", import.meta.url), "utf8", ).replace(/\s+/g, " "); +const responseCacheRetentionReconciliationMigration = readFileSync( + new URL("../supabase/migrations/20260713201542_consolidate_rag_response_cache_retention.sql", import.meta.url), + "utf8", +).replace(/\s+/g, " "); const liveDatabaseDriftMigration = readFileSync( new URL("../supabase/migrations/20260705230000_reconcile_live_database_drift.sql", import.meta.url), "utf8", @@ -1134,6 +1138,18 @@ describe("Supabase Preview replay guards", () => { expect(ragQueriesRetentionDuplicateMigration).toMatch(/select 1;/); }); + it("keeps response-cache cleanup bounded and consolidates its cron jobs", () => { + expect(responseCacheRetentionReconciliationMigration).toContain( + "where j.jobname in ('purge-rag-response-cache', 'purge-expired-rag-response-cache')", + ); + expect(responseCacheRetentionReconciliationMigration).toContain( + "$job$select public.purge_expired_rag_response_cache(1000);$job$", + ); + expect(responseCacheRetentionReconciliationMigration).not.toContain("delete from public.rag_response_cache"); + expect(schema).toContain("purge_expired_rag_response_cache(p_limit integer default 1000)"); + expect(schema).toContain("limit p_limit"); + }); + it("keeps ingestion_jobs_one_open stem neutralized for preview history parity", () => { expect(ingestionJobsOneOpenNeutralizedMigration).toContain("NEUTRALIZED 2026-07-09"); expect(ingestionJobsOneOpenNeutralizedMigration).toContain("select 1 where false;");