Harden RAG retrieval and answer reliability

docs/branch-review-ledger.md

@@ -1153,5 +1153,8 @@ This file is append-only. Never rewrite or delete an existing review record; app
 | 2026-07-27 | PR #1280 / `claude/top-search-design-mockups-w53znc` | `980b4298` | Implemented review follow-up | Synced main; rail overflow observes childList mutations. Temporarily disabled auto-merge to land polish without squash race. | Focused band Vitest 9/9; no provider checks. |
 | 2026-07-27 | PR #1281 / `claude/safety-planning-tools-page-tsq4vs` | `f7e616d1` | Implemented review polish | StepBuilderCard filled uses `isStepComplete`; clipboard DRAFT assertion added. Auto-merge temporarily disabled to land polish. | Focused safety-plan Vitest 3/3; no provider checks. |
 | 2026-07-27 | PR #1261/#1262/#1263 audit cluster | closed tips | Close without merge | Closed per review: unsafe lineage / tip markers / privacy+RAG P1s / parallel verify:cheap rewrite. Thin PDF exit-137 salvage opened separately. | Prior Bugbot + merge-tree evidence; no provider checks. |
+| 2026-07-28 | PR #1289 / `codex/rag-reliability-final` | `a1ca6a016490e4d4b564edd3d553d87fed3071df` | Protected-main RAG reliability, clinical-governance and release review | APPROVE. Independent retrieval, governance and fallback reviews found and fixed three merge blockers: global chunk-query alias overreach was narrowed to the measured clozapine blood-count action shape; legacy private source reviews remain on the deployed v1 RPC while unapplied-v2 paths fail explicitly; and source-backed review fallback is now a zero-tolerance blocking metric with reconciled evidence. Final rereviews found no P0-P2. PR #1288 was superseded without force-push after GitGuardian correctly rejected a token-shaped fake fixture; the clean replacement tree is byte-identical to the reviewed tree and both secret scanners pass. The additive BMJ attestation migration remains unapplied and BMJ stays unverified pending qualified human action. | Exact application tree `verify:pr-local` PASS: format, zero-warning lint, typecheck, 403 files and 4,101 tests passed with 2 skipped, production build/client-secret scan, and 36 offline RAG fixtures. Live 36-case canary PASS with document/content recall 1.0, zero failed cases and zero per-case document/content RR regressions; three cache-bypassed affected-path answer probes PASS with zero provider requests and zero generation cost. Earlier coverage PASS: 399 files, 4,062 passed and 2 skipped, RAG 86.83% statements and 90.79% lines. Hosted build, static, unit coverage, migration replay, Supabase Preview, Production UI, policy, Semgrep, Gitleaks and GitGuardian passed on the implementation tree; final evidence-only head requires the normal hosted rerun. |
+| 2026-07-28 | PR #1289 / `codex/rag-reliability-final` | `d49af8acb42ccedbfe6c8b3f03d30769ba906ec7` | Bugbot review (exact head) | APPROVE. No high-confidence P0–P2. Application `src/`/`supabase/`/`tests/`/`scripts/` trees are unchanged vs prior APPROVE tip `a1ca6a01`; tip delta is docs evidence + clean merge from `main`. No unresolved `cursor[bot]` Bugbot threads. Residual risk: unapplied BMJ attestation migration (`#022`) correctly fails closed with `503 source_review_v2_unavailable` until qualified hosted apply. | Focused high-risk Vitest 358 + 1502 passed; merge-tree CLEAN vs `origin/main`; no provider-backed checks; no PR comment mutations. |
 | 2026-07-27 | PR #1290 / `codex/search-performance-correctness-pr` | `82775e25fc1519c436a719b0d204a7c57332d811` | CI fix + Bugbot | Fixed P1 from trim commit: restored `sourceSearchInputRef` + double-rAF focus for mobile Search in document (was title-seeding). Cleared Prettier indent break that failed Static PR checks. Mergeable; 0 behind main; no unresolved review threads (Codex/CodeRabbit rate-limited). | Bugbot; vitest document-detail/private-access/universal-search/viewer-shell/audit-nav 186/186; maintainability 1734/1734; prettier check; no provider checks. |
 | 2026-07-28 | PR #1290 / `codex/search-performance-correctness-pr` | `3acf0ee3b6b1da10d6e0c76d20825d9eb0c76e48` | CI fix + Bugbot | Fixed P0 duplicate sourceSearchInputRef from tip 1e5ee645; restored sheet-safe Search-in-document focus; hardened openComposer with expectSingleSettledOwner for Production UI dual-composer race. Mergeable; 0 unresolved threads. | Bugbot; document-detail vitest 8/8; Playwright presentation/grouped typeahead 3/3; maintainability 1733/1734; no provider checks. |
+| 2026-07-28 | PR #1289 / `codex/rag-reliability-final` | `ab6ca036937bff1acaefbda8a5581d6d75f489b3` | Final current-main sync review | APPROVE pending fresh exact-head required checks. Merged current `origin/main` without conflict after its already-reviewed document-search and focus-path changes; no protected RAG, evaluation, migration, or RAG fixture surface changed from the live-canary application tree, and no P0-P2 finding remains. | `git merge-tree --write-tree` CLEAN before sync; branch-ledger guard and `git diff --check` PASS; prior exact application-tree `verify:pr-local` and live 36-case canary remain applicable; fresh hosted checks required. |

docs/evidence/rag-reliability-evidence-2026-07-27.md

@@ -0,0 +1,87 @@
+# RAG reliability evidence — 2026-07-27
+
+This record captures the final local/live evidence for the 2026-07-27 RAG reliability work. Raw eval
+artifacts remain under the gitignored `output/` directory; this file records their paths and the metrics
+needed to audit the backlog dispositions.
+
+## Protected retrieval canary
+
+- Final exact-head artifact: `output/rag-retrieval-post-exact-head.json`
+- Immediate comparison artifact: `output/rag-retrieval-post-final.json`
+- Tested application tree: `a1ca6a016490e4d4b564edd3d553d87fed3071df`
+- Cases: 36
+- Document recall@5: 1.0
+- Content recall@5: 1.0
+- Hit@K: 1.0
+- MRR@10: 0.8921 (unchanged from the immediate comparison)
+- Content MRR@10: 0.9406 (unchanged from the immediate comparison; original baseline 0.9389)
+- nDCG@10: 0.9308 (unchanged from the immediate comparison; original baseline 0.9276)
+- Irrelevant-at-10 rate: 0.0917 (unchanged)
+- Required-signal coverage: 1.0
+- Median/p90 latency: 19,729/55,660 ms (immediate comparison 13,563/56,765 ms)
+- Failed cases: 0
+- Latency-failed cases: 0
+- Per-case document/content reciprocal-rank regressions: 0
+
+The final protected retrieval change uses the already-ranked expanded clinical query for chunk selection
+only for the measured clozapine blood-count action shape: the raw query must identify clozapine plus a
+blood-count concept, and the ranked primary variant must carry the threshold/action terms. Discharge,
+generic red-section and other document lookups retain the raw user wording. This preserves FBC/WBC/ANC
+and withhold/stop aliases for the intended clozapine path without dropping unrelated raw intent. Ranking
+scores, comparator order, clamps and semantic reranking were not changed. The earlier artifact measured
+the broader intermediate query shape; the exact-head artifact above validates the narrowed final behavior.
+
+## Final 44-case answer gate
+
+- Artifact: `output/rag-quality-final9/retrieval-quality-2026-07-27T21-19-19-573Z.json`
+- Diagnostic dump: `output/rag-quality-final9-cases.json`
+- Cases: 44
+- Supported: 30/30 substantive and grounded
+- Unsupported: 14/14 correct
+- Citation failure rate: 0
+- Numeric-grounding failure rate: 0
+- Source-backed review fallbacks: 0
+- Route-ceiling failures: 0
+- Comparison extractive fallbacks: 2, both passed with distinct attributed sources
+- p95 latency: 7,494 ms
+- Provider use: 1 model case, 2 request IDs
+- Cost: unavailable because pricing-rate telemetry was not configured; no exact dollar value is claimed
+- Blocking threshold failures: none
+
+The final query-shape narrowing occurred after this 44-case artifact. The exact-head affected-path
+confirmations below therefore cover both admission/discharge cases and the clozapine threshold case; all
+other answer behavior remains covered by the full gate and the exact-head local test/build preflight.
+
+The immediately preceding `final8` run had the same perfect content metrics but one transient hosted
+retrieval tail: active-community ED took 15,760 ms against the 12,000 ms extractive ceiling. An isolated
+repeat passed at 8,100 ms and the final 44-case run passed at 9,610 ms. No latency ceiling was weakened.
+
+## Targeted live confirmations
+
+- Community home visits: two same-document AKG citations, substantive extractive answer, zero provider
+  calls, 4,812 ms.
+- Best Practice Prescription: two same-document AKG citations, substantive extractive answer, zero
+  provider calls, 1,560 ms.
+- Clozapine typo threshold: one complete NMHS red-range row, substantive extractive answer, zero
+  provider calls, 3,160 ms.
+- Clozapine ANC/FBC threshold: one complete NMHS red-range row, expected-file citation, zero provider
+  calls, 4,375 ms.
+- Discharge summary: source-backed extractive recovery, zero review fallback, 8,831 ms on the final
+  targeted probe.
+- Exact-head admission/discharge comparison: substantive grounded extractive answer, expected-source hit,
+  zero citation/numeric failures, zero provider calls and zero estimated cost, 4,302 ms.
+- Exact-head admission/discharge paraphrase: substantive grounded extractive answer, expected-source hit,
+  zero citation/numeric failures, zero provider calls and zero estimated cost, 7,315 ms.
+- Exact-head clozapine ANC/FBC threshold: substantive grounded extractive answer, expected-source hit,
+  zero citation/numeric failures, zero provider calls and zero estimated cost, 11,120 ms.
+
+## Deliberate boundaries
+
+- The prepared BMJ attestation migration was not applied to hosted Supabase. Qualified human review,
+  deliberate hosted apply/attestation and warning-rate remeasurement remain `#022`. Until that apply,
+  existing owner-scoped reviews keep using the legacy RPC and v2-only public/attestation requests fail
+  explicitly with `503 source_review_v2_unavailable`.
+- Firefox/WebKit scheduled evidence and human disposition of irrelevant-at-10 labels remain `#023`.
+- ADHD corpus/table accessibility and metabolic schedule evidence remain the two open parts of `#018`.
+- No hosted migration, source attestation, production-data write, ranking-score/comparator change,
+  grounding relaxation or citation-gate relaxation was performed as part of this evidence run.

docs/evidence/rag-top-local-review-manifest-2026-07-26.json

@@ -0,0 +1,205 @@
+{
+  "version": "top-local-review-evidence-v1",
+  "source_artifact": "github-actions/run-30216191889/eval-canary-output/golden-retrieval.json",
+  "source_generated_at": null,
+  "review_status": "pending_qualified_human_review",
+  "attestation_applied": false,
+  "entry_count": 10,
+  "entries": [
+    {
+      "key": "file:clozapine management by gp (nmhs).pdf",
+      "document_id": null,
+      "title": "Clozapine Management By GP(NMHS)",
+      "file_name": "Clozapine Management by GP (NMHS).pdf",
+      "publisher_code": "NMHS",
+      "jurisdiction": "Australia/WA",
+      "top_result_slots": 11,
+      "best_rank": 2,
+      "case_ids": [
+        "clozapine-anc-threshold",
+        "clozapine-cbc-abbreviation-threshold",
+        "monitoring-threshold-from-chart",
+        "show-source-table-image"
+      ],
+      "review_class": "local_wa",
+      "observed": {
+        "document_statuses": ["review_due"],
+        "clinical_validation_statuses": ["locally_reviewed"],
+        "extraction_qualities": ["good"]
+      },
+      "review_status": "pending_qualified_human_review",
+      "attestation_applied": false
+    },
+    {
+      "key": "file:alcohol and other drugs - addiction, toxicity and withdrawal (fsh).pdf",
+      "document_id": null,
+      "title": "Alcohol And Other Drugs - Addiction, Toxicity And Withdrawal(FSH)",
+      "file_name": "Alcohol and Other Drugs - Addiction, Toxicity and Withdrawal (FSH).pdf",
+      "publisher_code": "FSH",
+      "jurisdiction": "Australia/WA",
+      "top_result_slots": 6,
+      "best_rank": 1,
+      "case_ids": ["alcohol-ciwa-scoring", "alcohol-ciwa-threshold", "alcohol-withdrawal-management"],
+      "review_class": "local_wa",
+      "observed": {
+        "document_statuses": ["review_due"],
+        "clinical_validation_statuses": ["locally_reviewed"],
+        "extraction_qualities": ["good"]
+      },
+      "review_status": "pending_qualified_human_review",
+      "attestation_applied": false
+    },
+    {
+      "key": "file:mhatt assessment and treatment process (akg).pdf",
+      "document_id": null,
+      "title": "MHATT Assessment And Treatment Process(AKG)",
+      "file_name": "MHATT Assessment and Treatment Process (AKG).pdf",
+      "publisher_code": "AKG",
+      "jurisdiction": "Australia/WA",
+      "top_result_slots": 4,
+      "best_rank": 4,
+      "case_ids": ["schizophrenia-overview"],
+      "review_class": "local_wa",
+      "observed": {
+        "document_statuses": ["review_due"],
+        "clinical_validation_statuses": ["locally_reviewed"],
+        "extraction_qualities": ["good"]
+      },
+      "review_status": "pending_qualified_human_review",
+      "attestation_applied": false
+    },
+    {
+      "key": "file:clozapine coordinator and clozapine clinic (nmhs).pdf",
+      "document_id": null,
+      "title": "Clozapine Coordinator And Clozapine Clinic(NMHS)",
+      "file_name": "Clozapine Coordinator and Clozapine Clinic (NMHS).pdf",
+      "publisher_code": "NMHS",
+      "jurisdiction": "Australia/WA",
+      "top_result_slots": 3,
+      "best_rank": 1,
+      "case_ids": ["clozapine-anc-threshold", "clozapine-cbc-abbreviation-threshold", "show-source-table-image"],
+      "review_class": "local_wa",
+      "observed": {
+        "document_statuses": ["review_due"],
+        "clinical_validation_statuses": ["locally_reviewed"],
+        "extraction_qualities": ["good"]
+      },
+      "review_status": "pending_qualified_human_review",
+      "attestation_applied": false
+    },
+    {
+      "key": "file:women's and perinatal mental health referral and management guideline (kemh).pdf",
+      "document_id": null,
+      "title": "Women S And Perinatal Mental Health Referral And Management Guideline(KEMH)",
+      "file_name": "Women's and Perinatal Mental Health Referral and Management Guideline (KEMH).pdf",
+      "publisher_code": "KEMH",
+      "jurisdiction": "Australia/WA",
+      "top_result_slots": 3,
+      "best_rank": 7,
+      "case_ids": ["bipolar-management-summary", "bipolar-vs-schizoaffective", "vector-bipolar"],
+      "review_class": "local_wa",
+      "observed": {
+        "document_statuses": ["review_due"],
+        "clinical_validation_statuses": ["locally_reviewed"],
+        "extraction_qualities": ["good"]
+      },
+      "review_status": "pending_qualified_human_review",
+      "attestation_applied": false
+    },
+    {
+      "key": "file:mother baby unit admissions (fsh).pdf",
+      "document_id": null,
+      "title": "Mother Baby Unit Admissions(FSH)",
+      "file_name": "Mother Baby Unit Admissions (FSH).pdf",
+      "publisher_code": "FSH",
+      "jurisdiction": "Australia/WA",
+      "top_result_slots": 2,
+      "best_rank": 6,
+      "case_ids": ["bipolar-vs-schizoaffective"],
+      "review_class": "local_wa",
+      "observed": {
+        "document_statuses": ["review_due"],
+        "clinical_validation_statuses": ["locally_reviewed"],
+        "extraction_qualities": ["good"]
+      },
+      "review_status": "pending_qualified_human_review",
+      "attestation_applied": false
+    },
+    {
+      "key": "file:pressure injuries (akg).pdf",
+      "document_id": null,
+      "title": "Pressure Injuries(AKG)",
+      "file_name": "Pressure Injuries (AKG).pdf",
+      "publisher_code": "AKG",
+      "jurisdiction": "Australia/WA",
+      "top_result_slots": 2,
+      "best_rank": 6,
+      "case_ids": ["flowchart-next-step"],
+      "review_class": "local_wa",
+      "observed": {
+        "document_statuses": ["review_due"],
+        "clinical_validation_statuses": ["locally_reviewed"],
+        "extraction_qualities": ["good"]
+      },
+      "review_status": "pending_qualified_human_review",
+      "attestation_applied": false
+    },
+    {
+      "key": "file:referral to the mother baby unit mr202.09 (kemh).pdf",
+      "document_id": null,
+      "title": "Referral To The Mother Baby Unit MR202.09(KEMH)",
+      "file_name": "Referral to the Mother Baby Unit MR202.09 (KEMH).pdf",
+      "publisher_code": "KEMH",
+      "jurisdiction": "Australia/WA",
+      "top_result_slots": 2,
+      "best_rank": 8,
+      "case_ids": ["vector-postnatal"],
+      "review_class": "local_wa",
+      "observed": {
+        "document_statuses": ["unknown"],
+        "clinical_validation_statuses": ["unverified"],
+        "extraction_qualities": ["good"]
+      },
+      "review_status": "pending_qualified_human_review",
+      "attestation_applied": false
+    },
+    {
+      "key": "file:hampton house referral (fsh).pdf",
+      "document_id": null,
+      "title": "Hampton House Referral(FSH)",
+      "file_name": "Hampton House Referral (FSH).pdf",
+      "publisher_code": "FSH",
+      "jurisdiction": "Australia/WA",
+      "top_result_slots": 2,
+      "best_rank": 9,
+      "case_ids": ["bipolar-vs-schizoaffective"],
+      "review_class": "local_wa",
+      "observed": {
+        "document_statuses": ["review_due"],
+        "clinical_validation_statuses": ["locally_reviewed"],
+        "extraction_qualities": ["good"]
+      },
+      "review_status": "pending_qualified_human_review",
+      "attestation_applied": false
+    },
+    {
+      "key": "file:arousal and agitation drug management (camhs).pdf",
+      "document_id": null,
+      "title": "Arousal And Agitation Drug Management(CAMHS)",
+      "file_name": "Arousal and Agitation Drug Management (CAMHS).pdf",
+      "publisher_code": "CAMHS",
+      "jurisdiction": "Australia/WA",
+      "top_result_slots": 1,
+      "best_rank": 2,
+      "case_ids": ["medication-chart-dose-route"],
+      "review_class": "local_wa",
+      "observed": {
+        "document_statuses": ["review_due"],
+        "clinical_validation_statuses": ["locally_reviewed"],
+        "extraction_qualities": ["good"]
+      },
+      "review_status": "pending_qualified_human_review",
+      "attestation_applied": false
+    }
+  ]
+}