From 05efa25ff5920ad01ebe7c53f4c47c49dc324c8a Mon Sep 17 00:00:00 2001 From: BigSimmo <87357024+BigSimmo@users.noreply.github.com> Date: Wed, 12 Aug 2026 23:10:32 +0800 Subject: [PATCH 1/3] fix(rag): retry unverified numeric answers safely --- docs/outstanding-issues.md | 4 +- src/lib/rag/rag-extractive-answer.ts | 1 + src/lib/rag/rag-generation-failure.ts | 56 +++++++++++++++++ src/lib/rag/rag.ts | 91 +++++++++++++-------------- tests/rag-answer-fallback.test.ts | 72 ++++++++++++++++++--- tests/rag-generation-failure.test.ts | 45 +++++++++++++ 6 files changed, 211 insertions(+), 58 deletions(-) create mode 100644 src/lib/rag/rag-generation-failure.ts create mode 100644 tests/rag-generation-failure.test.ts diff --git a/docs/outstanding-issues.md b/docs/outstanding-issues.md index 3d8c5b7cff..ece608b132 100644 --- a/docs/outstanding-issues.md +++ b/docs/outstanding-issues.md @@ -55,7 +55,7 @@ removed after current-main verification; it is not missing recommended work. | ----: | -------------- | -------- | ------------------------------------------- | ------------------------------------------------------------------ | ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 1 | `#059` | A1 | Operator security + independent reviewer | Immediate approved security window | 1–3 hours plus verification | Verify every reported exposed credential (GitHub, OpenAI, Supabase service role/database, E2E) is retired; rotate anything still valid and update only intended secret stores. Never record values; stop before provider action without approval. | | 2 | `#053` | A1 | Operator — legal/privacy | Start now; finish before real patient use/privacy-approved release | 4–8 hours internal; 1–6 weeks elapsed | Execute DPAs; decide ZDR/residency; obtain cache behavior in writing; review subprocessors; obtain APP 8 and APP 5/1 counsel sign-off. Do not change public copy before approval. | -| 3 | `#231` | A1 | Specialist — answer path + Operator | Immediate approved live investigation | 2–4 hours plus provider | Live answers degrade to source-only when `answerRouteBudgetMs.fast` (25000) binds while retrieval is healthy. Measure and fix the fast-route budget / generation timeout; keep conservative source-only fallback. **Gate:** focused answer-route tests offline first; live probe only with explicit provider approval. **Stop:** do not weaken quality gates to hide timeouts; flag RAG surfaces before edit. | +| 3 | `#231` | A1 | Specialist — answer path + Operator | Provider-timeout isolation before any new canary | 30–90 minutes coding plus separately approved provider check | The bounded numeric-grounding repair remains green offline. The authorized current-budget canary used exactly one cache-bypassed application request and one provider request; it remained grounded with one citation and zero unverified numeric tokens, but `provider_timeout` returned the conservative extractive fallback after 21.267 seconds generation and 23.717 seconds total under the normal 25-second route budget (`route_deadline_exceeded=false`). This did not validate model synthesis. Next, isolate the provider-generation timeout without changing timeouts, retries, quality gates, or fallback caching; only then seek separate approval for a materially changed final canary. | | 4 | `#024` | A2 | High — browser/Next diagnostics | Provider-free macOS Safari host available | 1–2 hours | Reproduce document-source fallbacks in Safari/STP without Playwright interception; capture `_rsc` response evidence. Treat as an app defect only if native Safari reproduces; otherwise return to the harness. Never suppress `pageerror` or change CORS without proof. | | 5 | `#022` | A2 | Operator — clinical governance + Specialist | Policy implemented locally; hosted apply and human review pending | 1–2 hours apply; 0.5–1 day first ten | The auditable BMJ `third_party_reference_attested` policy, migration and top-ten evidence manifest are prepared without changing `clinical_validation_status=unverified`. A qualified operator must review evidence, apply the migration deliberately, attest eligible records, review the ten visible local documents, then remeasure warnings. | | 6 | `#023` | A2 | Specialist — RAG/browser diagnostics | After next weekly/manual matrix green (audit no longer blocks it) | 1–2 hours | Capture one Firefox/WebKit scheduled/manual datapoint and disposition the human irrelevant-at-10 labels. Matrix is structurally unblocked from blocking audit; do not spend on another RAG run. | @@ -227,7 +227,7 @@ removed after current-main verification; it is not missing recommended work. | #215 | P3 | task | Add image-optimization basics for lightbox, PWA lifecycle and demo PNGs | PARTIALLY DELIVERED, RE-MEASURED 2026-08-12. One of the three items is done: src/components/clinical-dashboard/image-lightbox.tsx now carries `decoding="async"` (note the path — the row's 'image-lightbox.tsx' is under clinical-dashboard/, and a search at the repo root finds nothing, which reads misleadingly as 'already fixed'). Still outstanding: src/components/pwa-lifecycle.tsx has NO `decoding="async"`; public/demo-documents/*.png are still PNG-only with no WebP/AVIF alternative; and SignedImage still has no `priority` prop for above-fold evidence images. Next: those three, in that order — the pwa-lifecycle attribute is a one-line change. Related image/perf work stays tracked under #016, #013, #117 and #147. | session 2026-08-02 /ledger sweep — docs/audit/performance-image-cwv-audit-2026-08-02.md | 2026-08-02 | | #221 | P3 | task | Local EmptyState, LoadingState and Chip duplicates still unconverged after PR-J | IN FLIGHT 2026-08-12 in PR #1841 and #1842 (cn()/LinkAction contracts and the EmptyState/Chip convergence). Checked against the open-PR list during the full ledger sweep. Do NOT start this row while that PR is open — duplicating a queued conversion is the exact failure #292 records, and it has happened twice. Re-verify this row against main after that PR merges, and close it there rather than here. PR-J converged what it could inside its allowlists and left four known duplicates, each blocked for a stated reason rather than missed. therapy-compass/ui.tsx defines its own LoadingState AND its own EmptyState used across nine screens (whole-module job, not a one-call-site conversion). mode-home-template.tsx ModeHomeStatusNotice is an EmptyState duplicate that four catalogue homes delegate to, which is why those four files show no diff. differentials-home.tsx has a local two-density Chip blocked by the cn() tailwind-merge gap. favourites-command-library-page.tsx SmallChip is driven by an eight-entry type-token map that Chip's five-tone vocabulary cannot express. Next action: take these as one convergence PR after the cn() decision lands, not piecemeal. Found during PR-J adoption, 2026-08-03. | session 2026-08-03 (PR-J Wave 5, Builder B) | 2026-08-02 | | #222 | P3 | task | Headers surface only partially converged in PR-J: mode-home-template and search-results-header-band untouched | VERIFIED CORRECT 2026-08-12 — re-checked against merged main and left open: Still unconverged: src/components/mode-home-template.tsx defines ModeHomeStatusNotice locally (:232) and imports neither PageHeader nor the DS EmptyState; search-results-header-band.tsx is likewise untouched. Note the adjacency — in-flight PR #1842 delegates ModeHomeStatusNotice to the DS EmptyState under #221, which is a different conversion from the PageHeader question this row asks. Re-check after #1842 merges. Builder A converged DsmPageHeader, InformationPageHeader and InformationPageBreadcrumbs onto PageHeader plus Breadcrumb, and declined two files with reasons. mode-home-template.tsx ModeHomeHero is a centred display hero on the fluid text-hero token and is the slot the in-flow phone composer sits in, so converging it onto a left-aligned PageHeader is a redesign of 13 mode homes that collides with the one-composer-per-page contract. search-results-header-band.tsx is a results spine carrying status, counts and filters, not a page-title stack, so its pin tests/search-results-header-band.dom.test.tsx remains unflipped. Both are defensible; both leave the headers surface partially adopted. Next action: decide whether either is in scope at all, or record them as permanently out of the PageHeader vocabulary. Found during PR-J adoption, 2026-08-03. | session 2026-08-03 (PR-J Wave 5, Builder A) | 2026-08-02 | -| #231 | P1 | issue | Generation fallbacks no longer stick in answer cache; lithium generation quality still falls back safely | PARTIAL 2026-08-12: This PR fixes the clinically consequential stale-fallback path: every answer whose routing or degraded reason contains generation_fallback is excluded from rag_response_cache. Offline evidence: 96 focused answer-route tests and 574 RAG fixture/contract tests passed. Approved live baseline/final canaries preserved 36/36 document and content recall at 1.0 with zero per-case reciprocal-rank regressions; the final 44-case answer gate had zero citation or numeric-grounding failures. A budget extension was tested and rejected: four cache-bypassed 'Lithium dosing?' probes remained grounded, cited safe extractive fallbacks at 35-40 second candidate budgets; the decisive 40-second probe completed generation in 25.272 seconds and 27.237 seconds total with route_deadline_exceeded=false, but failed generation quality. Therefore OPENAI_ANSWER_TIMEOUT_MS and the route budget are not the current residual binding cause. Next: instrument and reproduce the structured generation-quality failure using provider-safe metadata, then make a separate bounded output-quality fix with an offline fixture and live canary. Stop: do not increase route/provider timeouts or cache any generation fallback. | session 2026-08-04 (production triage, live /api/search + /api/answer) | 2026-08-04 | +| #231 | P1 | issue | Generation fallbacks no longer stick in answer cache; lithium generation quality still falls back safely | PARTIAL 2026-08-12: Generation fallbacks no longer stick in either answer cache. Approved baseline/final canaries preserved 36/36 document and content recall at 1.0 with zero per-case reciprocal-rank regressions; the final 44-case answer gate had zero citation or numeric-grounding failures. Route-budget and timeout increases remain ruled out. An explicitly approved 40-second diagnostic canary isolated numeric_faithfulness_gap after two model attempts while preserving a grounded one-citation extractive fallback. The bounded local repair promotes that existing numeric-verification signal into the existing fast-to-strong retry and supplies exact numeric-grounding instructions without adding a model attempt; its regression fixture rejects an unsupported 500 mg answer and returns only the source-supported 250 mg answer. Offline evidence remains green: focused failure/fallback proof 93/93, typecheck, and the full offline RAG contract 23/23 files and 575/575 tests with 36 golden cases. Final authorized current-budget canary: exactly one cache-bypassed application request and one provider request; grounded medium-confidence output with one citation and zero unverified numeric tokens, but provider_timeout caused the conservative extractive fallback after 21.267 seconds generation and 23.717 seconds total under the normal 25-second route budget; route_deadline_exceeded=false. This does not validate model-synthesis numeric repair, so retain the safe fallback and keep the item open. Next: do not rerun immediately; isolate the provider-generation timeout without increasing timeouts or retries, then seek separate approval only for a materially changed final canary. Stop: do not expose provider output, weaken numeric or quality gates, add retries, increase route/provider timeouts, or cache any generation fallback. | session 2026-08-04 (production triage, live /api/search + /api/answer) | 2026-08-04 | | #233 | P3 | task | COMPONENTS.md section 0 describes the pre-adoption world, and the optionality-marker contract change is undocumented | IN FLIGHT 2026-08-12 in PR #1842 (records DS adoption evidence and convergence state). Checked against the open-PR list during the full ledger sweep. Do NOT start this row while that PR is open — duplicating a queued conversion is the exact failure #292 records, and it has happened twice. Re-verify this row against main after that PR merges, and close it there rather than here. Two documentation debts left by PR-J, both in docs/design-system/COMPONENTS.md, naturally one PR. First: section 0's maturity matrix is stale. FormField, TextField, SearchField, Select, Checkbox, RadioGroup, PageHeader and Breadcrumb now have real product mounts, so 0.1 and 0.2 misdescribe what is registered versus built-but-unregistered, and 0.4's field-shell defects are closed by the five-control fold. A reader deciding whether a component is safe to adopt is reading the wrong answer. Second: FormField now marks only the requirement and leaves optional fields unmarked - (optional) was removed app-wide by design decision and is pinned by tests/ui-v2-form-field.dom.test.tsx - which is a design-system contract change that appears in no document. It belongs in COMPONENTS.md section 4 and probably DECISIONS.md. Next action: one docs PR updating section 0 from the actual mount list and recording the optionality rule with its rationale. Stop: do not re-add (optional) markers to satisfy a generic form-accessibility rule - the removal was deliberate and is test-pinned. | session 2026-08-04 (DS V2 Wave 5 close-out capture) | 2026-08-04 | | #234 | P3 | task | answer-copy-payload.ts is the single clipboard payload builder for three surfaces and has no documentation | IN FLIGHT 2026-08-12 in PR #1842 (publishes the answer-copy clipboard contract). Checked against the open-PR list during the full ledger sweep. Do NOT start this row while that PR is open — duplicating a queued conversion is the exact failure #292 records, and it has happened twice. Re-verify this row against main after that PR merges, and close it there rather than here. src/lib/answer-copy-payload.ts arrived in PR-J exporting answerStateForAnswer, buildAnswerClipboardText, resolveAnswerSources, citedSourcesOnly and singleDocumentClipboardMetadata. It is now the one place three product surfaces build a clipboard payload, which makes it a contract rather than a helper: a future caller that bypasses it can reintroduce the false-attribution defect the module exists to prevent (see #228). Nothing in docs/design-system mentions it. Next action: document the module and its five exports where the answer surface's copy contract is described, and state that new copy paths route through it rather than composing their own text. Found during PR-J close-out, 2026-08-04. | session 2026-08-04 (DS V2 Wave 5 close-out capture) | 2026-08-04 | | #235 | P3 | task | ADOPTION.md section 7 proof shots exist for only four of the adopted surfaces | IN FLIGHT 2026-08-12 in PR #1842 (records adoption evidence). Checked against the open-PR list during the full ledger sweep. Do NOT start this row while that PR is open — duplicating a queued conversion is the exact failure #292 records, and it has happened twice. Re-verify this row against main after that PR merges, and close it there rather than here. The adoption contract asks for a proof shot per adopted surface. The Wave 5 adoption captured four - DSM header, settings rows, patient panel, answer surface - and none for the forms fold, the catalogue and docs surfaces, the headers convergence, or the empty states adopted since. Section 7 therefore reads as complete while most of the adoption is unevidenced, which matters because the proof shot is what a later reader uses to tell an intended restyle from a regression (the #229 DSM eyebrow was almost rediscovered as a defect for exactly this reason). Next action: capture the missing shots against a warmed local server and attach them to section 7. Cheap and mechanical - no gate, no provider access. Stop: this is not the visual-baseline harness (#118) - do not commit Playwright snapshot PNGs or flip that job to blocking. | session 2026-08-04 (DS V2 Wave 5 close-out capture) | 2026-08-04 | diff --git a/src/lib/rag/rag-extractive-answer.ts b/src/lib/rag/rag-extractive-answer.ts index a9de572dd9..df02664b73 100644 --- a/src/lib/rag/rag-extractive-answer.ts +++ b/src/lib/rag/rag-extractive-answer.ts @@ -3276,6 +3276,7 @@ export function generatedAnswerQualityFailureReason(answer: RagAnswer, query: st return "missing_query_overlap"; } if (hasInvalidModelEvidenceIds(answer)) return "invalid_model_evidence_ids"; + if (answer.unverifiedNumericTokens?.length) return "numeric_faithfulness_gap"; const broadDocumentCoverageRequested = queryClass === "document_lookup" && /(?:\b(?:what|which)\b.{0,100}\b(?:include|included|require|required|requirements?)\b|\b(?:process|procedure)\b|\bhow\b.{0,80}\b(?:handled|managed|performed|completed)\b)/i.test( diff --git a/src/lib/rag/rag-generation-failure.ts b/src/lib/rag/rag-generation-failure.ts new file mode 100644 index 0000000000..b67dbde5f9 --- /dev/null +++ b/src/lib/rag/rag-generation-failure.ts @@ -0,0 +1,56 @@ +const PROVIDER_SAFE_GENERATION_QUALITY_FAILURE_REASONS = new Set([ + "empty_after_sanitize", + "provider_source_gap", + "incomplete_opening_sentence", + "bad_final_answer_quality", + "clinical_answer_quality_issue", + "low_yield_answer", + "fragment_like_answer", + "missing_query_intent", + "missing_query_overlap", + "invalid_model_evidence_ids", + "insufficient_broad_citation_coverage", + "unusable_generated_answer", + "template_like_answer", + "overexpanded_simple_answer", + "claim_support_high_risk_gap", + "material_source_governance_gap", + "numeric_band_coherence_gap", + "numeric_faithfulness_gap", +]); + +/** Reduce generation errors to bounded, provider-safe diagnostic metadata. */ +export function summarizeGenerationFailureReason(error: unknown) { + const message = (error instanceof Error ? error.message : typeof error === "string" ? error : "").trim(); + const normalized = message.toLowerCase(); + const sourceBackedRecovery = normalized.match(/\bsource_backed_extractive_recovery:([a-z0-9_]+)/); + + if (sourceBackedRecovery) return `source_backed_extractive_recovery_${sourceBackedRecovery[1]}`; + if (!normalized) return "generation_failed"; + if (/\bprovider_source_gap\b/.test(normalized)) return "provider_source_gap"; + + const qualityFailure = normalized.match(/^openai generation quality gate failed:\s*([a-z0-9_]+)$/); + if (qualityFailure && PROVIDER_SAFE_GENERATION_QUALITY_FAILURE_REASONS.has(qualityFailure[1])) { + return `generation_quality_failed_${qualityFailure[1]}`; + } + if (normalized.startsWith("openai generation quality gate failed:")) return "generation_quality_failed"; + + if (/\bmax_output_tokens\b/.test(normalized)) return "provider_incomplete_max_output_tokens"; + if (/\bincomplete\b/.test(normalized)) return "provider_incomplete"; + if (/\brate limit|rate_limited|429\b/.test(normalized)) return "provider_rate_limited"; + if (/\btimeout|timed out|deadline|aborted|etimedout\b/.test(normalized)) return "provider_timeout"; + if (/\bauthentication|api key|unauthori[sz]ed|401|403\b/.test(normalized)) return "provider_auth_failed"; + + if (/\bvalidation|quality gate|schema|parse|json\b/.test(normalized)) return "generation_quality_failed"; + if (/\bopenai|provider|model\b/.test(normalized)) return "provider_generation_failed"; + return "generation_failed"; +} + +/** Build the bounded repair instruction for an existing strong-model retry. */ +export function generationQualityRetryInstruction(failureReason: string) { + const numericFaithfulnessInstruction = + failureReason === "numeric_faithfulness_gap" + ? " The previous answer included a numeric token that deterministic verification could not match to its cited retrieved evidence. Include a number, dose, frequency, threshold, or timing only when its exact digits and unit appear in the cited source excerpt; otherwise omit it. Do not infer, convert, calculate, round, or combine figures." + : ""; + return `The previous answer failed deterministic validation (${failureReason}).${numericFaithfulnessInstruction} Return schema-valid output only, with a complete natural clinical synthesis in the answer field. The first sentence must directly answer the question as a full sentence. Every clinical claim must be supported by valid retrieved citation_chunk_id values; do not invent citation IDs. Within one named scale and source, if differently labelled intervals overlap or a range is reversed, omit the entire affected band set; do not quote, repair, or infer any label or value. If a separate sentence or clause states a nonnumeric condition and action independent of the score, answer only with that independently supported condition and action, cite the smallest sufficient directly supporting chunk set, and add a conflict entry; otherwise return a source gap. Avoid template/source-inventory wording and do not include JSON fragments inside text fields. If the evidence cannot support the requested clinical answer, return a concise source-gap answer instead. If the question is a simple definition or direct fact question, answer only that question and return answerSections as an empty array unless a source-gap or safety caveat is essential.`; +} diff --git a/src/lib/rag/rag.ts b/src/lib/rag/rag.ts index debe71a516..0edf9f52b2 100644 --- a/src/lib/rag/rag.ts +++ b/src/lib/rag/rag.ts @@ -36,6 +36,7 @@ import { isSourceOnlyMode, sourceOnlyReason, } from "@/lib/rag/rag-provider"; +import { generationQualityRetryInstruction, summarizeGenerationFailureReason } from "@/lib/rag/rag-generation-failure"; import { allowedChunkMap, citationFromResult as resultCitation, compactCitations } from "@/lib/citations"; import { assessAndEnforceClaimSupport, enforceLabelledNumericBandCoherence } from "@/lib/rag/rag-claim-support"; import { @@ -3375,6 +3376,7 @@ ${qualityRetryInstruction}` "fast_unusable_retry_strong", "fast_template_retry_strong", "fast_quality_retry_strong", + "fast_numeric_faithfulness_retry_strong", ]); const eligibleForRoutineExtractiveRecovery = route.mode === "fast" && @@ -3399,25 +3401,6 @@ ${qualityRetryInstruction}` }); } - /** Summarize generation failure reason. */ - function summarizeGenerationFailureReason(error: unknown) { - const message = (error instanceof Error ? error.message : typeof error === "string" ? error : "").trim(); - const normalized = message.toLowerCase(); - const sourceBackedRecovery = normalized.match(/\bsource_backed_extractive_recovery:([a-z0-9_]+)/); - - if (sourceBackedRecovery) return `source_backed_extractive_recovery_${sourceBackedRecovery[1]}`; - if (!normalized) return "generation_failed"; - if (/\bprovider_source_gap\b/.test(normalized)) return "provider_source_gap"; - if (/\bmax_output_tokens\b/.test(normalized)) return "provider_incomplete_max_output_tokens"; - if (/\bincomplete\b/.test(normalized)) return "provider_incomplete"; - if (/\brate limit|rate_limited|429\b/.test(normalized)) return "provider_rate_limited"; - if (/\btimeout|timed out|deadline|aborted|etimedout\b/.test(normalized)) return "provider_timeout"; - if (/\bauthentication|api key|unauthori[sz]ed|401|403\b/.test(normalized)) return "provider_auth_failed"; - if (/\bvalidation|quality gate|schema|parse|json\b/.test(normalized)) return "generation_quality_failed"; - if (/\bopenai|provider|model\b/.test(normalized)) return "provider_generation_failed"; - return "generation_failed"; - } - /** Build generation fallback answer. */ async function buildGenerationFallbackAnswer( error: unknown, @@ -3598,12 +3581,13 @@ ${qualityRetryInstruction}` !fastSourceGap && !fastAnswerWasTemplateLike && shouldRetryWithStrongAfterFast({ route, answer, results: answerInputResults }); - const fastAnswerFailedQualityGate = + const fastQualityFailureReason = route.mode === "fast" && !fastAnswerWasUnusable && !fastAnswerWasTemplateLike && !fastAnswerWasOverExpanded && - Boolean(generatedAnswerQualityFailureReason(answer, args.query, queryClass)); + generatedAnswerQualityFailureReason(answer, args.query, queryClass); + const fastAnswerFailedQualityGate = Boolean(fastQualityFailureReason); if ( fastAnswerHadInvalidEvidenceIds || fastSourceGap || @@ -3613,19 +3597,22 @@ ${qualityRetryInstruction}` fastAnswerWasOverExpanded || fastAnswerFailedQualityGate ) { - const retryReason = fastAnswerHadInvalidEvidenceIds - ? "fast_invalid_evidence_retry_strong" - : fastSourceGap - ? "fast_source_gap_retry_strong" - : fastAnswerWasUnsupported - ? "fast_unsupported_retry_strong" - : fastAnswerWasUnusable - ? "fast_unusable_retry_strong" - : fastAnswerWasTemplateLike - ? "fast_template_retry_strong" - : fastAnswerWasOverExpanded - ? "fast_overexpanded_simple_retry_strong" - : "fast_quality_retry_strong"; + const retryReason = + fastQualityFailureReason === "numeric_faithfulness_gap" + ? "fast_numeric_faithfulness_retry_strong" + : fastAnswerHadInvalidEvidenceIds + ? "fast_invalid_evidence_retry_strong" + : fastSourceGap + ? "fast_source_gap_retry_strong" + : fastAnswerWasUnsupported + ? "fast_unsupported_retry_strong" + : fastAnswerWasUnusable + ? "fast_unusable_retry_strong" + : fastAnswerWasTemplateLike + ? "fast_template_retry_strong" + : fastAnswerWasOverExpanded + ? "fast_overexpanded_simple_retry_strong" + : "fast_quality_retry_strong"; if (shouldRecoverFastFailureExtractively(retryReason)) { answerRetryCount += 1; answerRetryReasons.push(`fast_source_backed_extractive_recovery:${retryReason}`); @@ -3639,19 +3626,21 @@ ${qualityRetryInstruction}` await args.onProgress?.({ stage: "retrying", message: - retryReason === "fast_invalid_evidence_retry_strong" - ? "Fast answer cited invalid evidence IDs, retrying with the strong model." - : retryReason === "fast_source_gap_retry_strong" - ? "Fast answer returned a source gap despite strong retrieval, retrying with the strong model." - : retryReason === "fast_unsupported_retry_strong" - ? "Fast answer was unsupported, retrying with the strong model." - : retryReason === "fast_unusable_retry_strong" - ? "Fast answer was not usable, retrying with the strong model." - : retryReason === "fast_template_retry_strong" - ? "Fast answer was too template-like, retrying with the strong model." - : retryReason === "fast_overexpanded_simple_retry_strong" - ? "Fast answer over-expanded a simple question, retrying with the strong model." - : "Fast answer failed quality checks, retrying with the strong model.", + retryReason === "fast_numeric_faithfulness_retry_strong" + ? "Fast answer included an unverified figure, retrying with exact numeric-grounding instructions." + : retryReason === "fast_invalid_evidence_retry_strong" + ? "Fast answer cited invalid evidence IDs, retrying with the strong model." + : retryReason === "fast_source_gap_retry_strong" + ? "Fast answer returned a source gap despite strong retrieval, retrying with the strong model." + : retryReason === "fast_unsupported_retry_strong" + ? "Fast answer was unsupported, retrying with the strong model." + : retryReason === "fast_unusable_retry_strong" + ? "Fast answer was not usable, retrying with the strong model." + : retryReason === "fast_template_retry_strong" + ? "Fast answer was too template-like, retrying with the strong model." + : retryReason === "fast_overexpanded_simple_retry_strong" + ? "Fast answer over-expanded a simple question, retrying with the strong model." + : "Fast answer failed quality checks, retrying with the strong model.", mode: "strong", model: env.OPENAI_STRONG_ANSWER_MODEL, reason: routingReason, @@ -3663,6 +3652,10 @@ ${qualityRetryInstruction}` generated = await generateWithModel(env.OPENAI_STRONG_ANSWER_MODEL, packedContextResults, { strong: true, maxOutputTokensOverride: strongRetryMaxOutputTokens, + qualityRetryInstruction: + fastQualityFailureReason === "numeric_faithfulness_gap" + ? generationQualityRetryInstruction(fastQualityFailureReason) + : undefined, }); retrievalDiagnostics.routeMode = "strong"; if (generated.truncated) { @@ -3697,7 +3690,7 @@ ${qualityRetryInstruction}` // valid (if imperfect) cited strong answer instead of spending a third generation // and risking a truncation -> unsupported tail. Recorded for observability. answerRetryReasons.push(`strong_quality_repair_skipped_time_budget:${strongQualityFailureReason}`); - } else if (answerNeedsStrongQualityRepair) { + } else if (strongQualityFailureReason) { routingReason = `${routingReason}; strong_quality_retry`; answerRetryCount += 1; answerRetryReasons.push("strong_quality_retry"); @@ -3711,7 +3704,7 @@ ${qualityRetryInstruction}` generated = await generateWithModel(env.OPENAI_STRONG_ANSWER_MODEL, packedContextResults, { strong: true, maxOutputTokensOverride: strongRetryMaxOutputTokens, - qualityRetryInstruction: `The previous answer failed deterministic validation (${strongQualityFailureReason}). Return schema-valid output only, with a complete natural clinical synthesis in the answer field. The first sentence must directly answer the question as a full sentence. Every clinical claim must be supported by valid retrieved citation_chunk_id values; do not invent citation IDs. Within one named scale and source, if differently labelled intervals overlap or a range is reversed, omit the entire affected band set; do not quote, repair, or infer any label or value. If a separate sentence or clause states a nonnumeric condition and action independent of the score, answer only with that independently supported condition and action, cite the smallest sufficient directly supporting chunk set, and add a conflict entry; otherwise return a source gap. Avoid template/source-inventory wording and do not include JSON fragments inside text fields. If the evidence cannot support the requested clinical answer, return a concise source-gap answer instead. If the question is a simple definition or direct fact question, answer only that question and return answerSections as an empty array unless a source-gap or safety caveat is essential.`, + qualityRetryInstruction: generationQualityRetryInstruction(strongQualityFailureReason), }); retrievalDiagnostics.routeMode = "strong"; if (generated.truncated) { diff --git a/tests/rag-answer-fallback.test.ts b/tests/rag-answer-fallback.test.ts index 5d05302bbe..77cffa5018 100644 --- a/tests/rag-answer-fallback.test.ts +++ b/tests/rag-answer-fallback.test.ts @@ -91,8 +91,8 @@ type GeneratedAnswerPayload = { async function answerFromTextSources( query: string, sources: SearchResult[], - generatedAnswer?: GeneratedAnswerPayload | Error, - options: { sourceOnly?: boolean } = {}, + generatedAnswer?: GeneratedAnswerPayload | Error | Array, + options: { sourceOnly?: boolean; onGenerate?: (input: string, index: number) => void } = {}, ) { // `src/lib/env.ts` freezes process.env at module load. The offline vitest wrapper // starts every worker as RAG_PROVIDER_MODE=offline with a blank OpenAI key, so we @@ -116,11 +116,16 @@ async function answerFromTextSources( from: vi.fn(() => new EmptyQuery()), }), })); - const generateStructuredTextResult = vi.fn(async () => { - if (generatedAnswer instanceof Error) throw generatedAnswer; + const generatedAnswers = Array.isArray(generatedAnswer) ? generatedAnswer : [generatedAnswer]; + let generatedAnswerIndex = 0; + const generateStructuredTextResult = vi.fn(async (input: string) => { + options.onGenerate?.(input, generatedAnswerIndex); + const currentGeneratedAnswer = generatedAnswers[Math.min(generatedAnswerIndex, generatedAnswers.length - 1)]; + generatedAnswerIndex += 1; + if (currentGeneratedAnswer instanceof Error) throw currentGeneratedAnswer; return { text: JSON.stringify( - generatedAnswer ?? { + currentGeneratedAnswer ?? { answer: "No current source with specific guidance for this query was found.", grounded: false, confidence: "unsupported", @@ -165,6 +170,59 @@ afterEach(() => { }); describe("RAG structured-output fallback", () => { + it("repairs an unsupported fast-answer figure with the existing strong retry", async () => { + const answer = await answerFromTextSources( + "Lithium dosing?", + [ + source({ + id: "lithium-dose-source", + document_id: "lithium-guideline", + title: "Medication guideline", + file_name: "medication-guideline.pdf", + section_heading: "Lithium initiation", + content: "Start lithium carbonate at 250 mg once daily and review tolerability.", + similarity: 0.94, + hybrid_score: 0.94, + text_rank: 0.09, + }), + ], + [ + { + answer: "Start lithium carbonate at 500 mg once daily.", + grounded: true, + confidence: "high", + answerSections: [], + citations: [{ chunk_id: "lithium-dose-source" }], + quoteCards: [], + conflictsOrGaps: [], + }, + { + answer: "Start lithium carbonate at 250 mg once daily.", + grounded: true, + confidence: "high", + answerSections: [], + citations: [{ chunk_id: "lithium-dose-source" }], + quoteCards: [], + conflictsOrGaps: [], + }, + ], + { + onGenerate: (input, index) => { + if (index !== 1) return; + expect(input).toContain("numeric_faithfulness_gap"); + expect(input).toContain("exact digits and unit appear in the cited source excerpt"); + }, + }, + ); + + expect(answer.routingMode).toBe("strong"); + expect(answer.routingReason).toContain("fast_numeric_faithfulness_retry_strong"); + expect(answer.routingReason).not.toContain("generation_fallback"); + expect(answer.answer.replace(/\*\*/g, "")).toContain("250 mg"); + expect(answer.unverifiedNumericTokens ?? []).toEqual([]); + expect(answer.openAIRequestIds).toEqual(["req_answer_from_text_sources", "req_answer_from_text_sources"]); + }); + it("recovers a cited provider source gap instead of treating nearby citations as a grounded answer", async () => { const dischargeSources = [ source({ @@ -1510,13 +1568,13 @@ describe("RAG structured-output fallback", () => { text_rank: 1, }), ], - new Error("OpenAI generation quality gate failed: labelled numeric band conflict"), + new Error("OpenAI generation quality gate failed: numeric_band_coherence_gap"), ); const deliveredText = `${answer.answer} ${(answer.answerSections ?? []).map((section) => section.body).join(" ")}`.replace(/\*\*/g, ""); expect(answer.grounded).toBe(true); - expect(answer.routingReason).toContain("generation_fallback:generation_quality_failed"); + expect(answer.routingReason).toContain("generation_fallback:generation_quality_failed_numeric_band_coherence_gap"); expect(answer.routingReason).toContain("source_backed_extractive_fallback"); expect(deliveredText).toMatch( /any side effect which is causing distress irrespective of score should be escalated to the treating doctor and reviewed/i, diff --git a/tests/rag-generation-failure.test.ts b/tests/rag-generation-failure.test.ts new file mode 100644 index 0000000000..d3b028e374 --- /dev/null +++ b/tests/rag-generation-failure.test.ts @@ -0,0 +1,45 @@ +import { describe, expect, it } from "vitest"; +import { + generationQualityRetryInstruction, + summarizeGenerationFailureReason, +} from "../src/lib/rag/rag-generation-failure"; + +describe("generation failure diagnostics", () => { + it("preserves an allowlisted deterministic quality-gate reason", () => { + expect( + summarizeGenerationFailureReason(new Error("OpenAI generation quality gate failed: missing_query_intent")), + ).toBe("generation_quality_failed_missing_query_intent"); + }); + + it("preserves an allowlisted post-finalization quality-gate reason", () => { + expect( + summarizeGenerationFailureReason(new Error("OpenAI generation quality gate failed: numeric_faithfulness_gap")), + ).toBe("generation_quality_failed_numeric_faithfulness_gap"); + }); + + it("does not confuse an incomplete-opening quality gate with provider truncation", () => { + expect( + summarizeGenerationFailureReason(new Error("OpenAI generation quality gate failed: incomplete_opening_sentence")), + ).toBe("generation_quality_failed_incomplete_opening_sentence"); + }); + + it("does not expose an unknown quality-gate message", () => { + expect( + summarizeGenerationFailureReason( + new Error("OpenAI generation quality gate failed: incomplete private patient details"), + ), + ).toBe("generation_quality_failed"); + }); + + it("keeps provider failure classification unchanged", () => { + expect(summarizeGenerationFailureReason(new Error("OpenAI provider timed out"))).toBe("provider_timeout"); + }); + + it("adds a bounded numeric-faithfulness repair instruction", () => { + const instruction = generationQualityRetryInstruction("numeric_faithfulness_gap"); + + expect(instruction).toContain("exact digits and unit appear in the cited source excerpt"); + expect(instruction).toContain("Do not infer, convert, calculate, round, or combine figures"); + expect(instruction).toContain("numeric_faithfulness_gap"); + }); +}); From 95a642ce10a5377cf41b7fcaf13dff86d8358689 Mon Sep 17 00:00:00 2001 From: BigSimmo <87357024+BigSimmo@users.noreply.github.com> Date: Wed, 12 Aug 2026 23:52:45 +0800 Subject: [PATCH 2/3] docs(ledger): record RAG numeric repair review --- docs/branch-review-ledger.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/branch-review-ledger.md b/docs/branch-review-ledger.md index 5df0ad13fc..ee0515bcf0 100644 --- a/docs/branch-review-ledger.md +++ b/docs/branch-review-ledger.md @@ -914,3 +914,4 @@ Records before 2026-07-28 were written by hand and had drifted: 146 lines carrie | 2026-08-12 | PR-1836 | eef2e275b7d5944206f1a27e23fdd78d95b383d8 | full PR diff and unresolved review feedback | No P0-P2 findings; prior review feedback already fixed and dispositioned | focused ledger/docs/workflow guards pass after current-main merge | | 2026-08-12 | claude/segmented-control-count | d5ff3160242986d2fac80ed9a915390b5c69b6e1 | SegmentedControl option hint slot (filter rollout prerequisite) | PR #1848 opened; additive, no call site passes hint yet; fixed a concatenated accessible name (All62 -> All (62)) | lint/typecheck/test green, 80 in ui-v2-components, 56 design-sync with no regen needed, clean build, bundle-budget within tolerance | | 2026-08-12 | 1848 | c65b9d91b760862c2ff9d5001974670219c5da02 | full PR diff and unresolved review feedback | P2 hint contrast and live-count width fixes applied | focused Vitest passed (81); focused ESLint passed; design-system contract passed | +| 2026-08-12 | codex/rag-numeric-grounding-231 | 05efa25ff5920ad01ebe7c53f4c47c49dc324c8a | answer-generation numeric-grounding repair and #231 canary ledger evidence | No high-confidence diff finding; draft handoff appropriate. Current-main lint/typecheck and static guards passed. Full suite failed on 20 unrelated Windows/contention/baseline tests before build/offline RAG; focused rerun was coordinator-blocked twice. Prior identical implementation passed 93/93 focused and 575/575 offline RAG tests. Live canary retained safe fallback but provider timeout prevented synthesis validation. | verify:pr-local partial; prior focused 93/93; prior offline RAG 575/575; authorized one-shot canary source-only fallback | From d911c4b3fe1f06dd8e5bae661bbda603fe9673e1 Mon Sep 17 00:00:00 2001 From: BigSimmo <87357024+BigSimmo@users.noreply.github.com> Date: Wed, 12 Aug 2026 23:59:38 +0800 Subject: [PATCH 3/3] docs(ledger): supersede RAG review after main sync --- docs/branch-review-ledger.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/branch-review-ledger.md b/docs/branch-review-ledger.md index 60113e875d..062d9c2613 100644 --- a/docs/branch-review-ledger.md +++ b/docs/branch-review-ledger.md @@ -916,3 +916,4 @@ Records before 2026-07-28 were written by hand and had drifted: 146 lines carrie | 2026-08-12 | 1848 | c65b9d91b760862c2ff9d5001974670219c5da02 | full PR diff and unresolved review feedback | P2 hint contrast and live-count width fixes applied | focused Vitest passed (81); focused ESLint passed; design-system contract passed | | 2026-08-12 | codex/rag-numeric-grounding-231 | 05efa25ff5920ad01ebe7c53f4c47c49dc324c8a | answer-generation numeric-grounding repair and #231 canary ledger evidence | No high-confidence diff finding; draft handoff appropriate. Current-main lint/typecheck and static guards passed. Full suite failed on 20 unrelated Windows/contention/baseline tests before build/offline RAG; focused rerun was coordinator-blocked twice. Prior identical implementation passed 93/93 focused and 575/575 offline RAG tests. Live canary retained safe fallback but provider timeout prevented synthesis validation. | verify:pr-local partial; prior focused 93/93; prior offline RAG 575/575; authorized one-shot canary source-only fallback | | 2026-08-12 | claude/filter-lens-modes | 5cf8871ef122ec9e3d9b25ea65c4643d5a2cf2ba | filter contract PR A: lens adoption across differentials, medication, applications, specifiers | PR #1857 opened; 4 bespoke aria-pressed rails converged onto SegmentedControl with one shared option array per mode; specifiers footerNote fixed (counted results+catalogueMatches while filters govern only results); ResultFilterSheet counted-option accessible name fixed (All8 -> All (8)) on both group kinds; SegmentedControl gained group-level ariaControls so the launcher keeps #launcher-results-panel; dead SpecifierFamilyFilterChips removed; scope segment deliberately deferred to services per filter-contract.md s4 | verify:pr-local all steps green except build, which failed on the /issues #210 dev-types corruption and passed on a clean rebuild; unit suite 6100 passed/4 skipped with one 30s timeout (not an assertion failure) in design-sync-contract under parallel load, passing in isolation 7/7; bundle-budget production 1297.7 KiB and mockups 285.1 KiB both within tolerance on a verified-fresh build; browser proof at 1440/800/390/320px on all four modes, 0px overflow, 48px targets | +| 2026-08-12 | codex/rag-numeric-grounding-231 | ba363cc3cd1095f5cc6edd1753cc722ed40ce518 | answer-generation numeric-grounding repair and #231 canary ledger evidence | Supersedes pre-main-sync review record; target seven-file PR diff is unchanged after merging origin/main. No high-confidence diff finding; draft handoff appropriate. Current-main lint/typecheck and static guards passed. Full suite failed on 20 unrelated Windows/contention/baseline tests before build/offline RAG; focused rerun was coordinator-blocked twice. Prior identical implementation passed 93/93 focused and 575/575 offline RAG tests. Live canary retained safe fallback but provider timeout prevented synthesis validation. | verify:pr-local partial; prior focused 93/93; prior offline RAG 575/575; authorized one-shot canary source-only fallback |