feat(answer): compose follow-up chips from the S2 menu, gate them on evidence (packet S3 / A4) - #2108
Conversation
…evidence (packet S3 / A4) buildAnswerFollowUpSuggestions previously offered the same four templated questions for every medication answer, keyed off a regex over the question text. It could name a subject the corpus never surfaced, and it asked again for something the answer had already given. Refine it in place per docs/rag-improvement/README.md A4: - candidates come from the S2 related-information menu (buildRelatedInformationMenu) for the answer's own query class and intent, with one deterministic question per menu item, index-aligned with the menu; - each candidate must be supported by the retrieved evidence the client actually received (bounded source snippets, quote cards, safety findings). Query-derived canonical terms vouch for the SUBJECT only, never for corpus coverage, so the gate can under-suggest but never over-suggest; - a kind the answer already emitted as an answerSections entry, or whose concept words are already in the answer body, is suppressed. When nothing survives, no chip row renders; - classes whose menu is "none" keep the previous narrow templates, now gated and suppressed the same way; - the subject is read back out of the clinician's question, so acronyms keep their casing and an answer-derived medication no longer hijacks the subject of an unrelated question. The four-chip cap, thread de-duplication and deterministic phrasing are unchanged; there is no provider call and no new render path. Nothing under src/lib/rag/** is edited - answer-composition.ts is only imported. ClinicalDashboard.tsx is unchanged: queryClass and queryAnalysis.intent already ride on RagAnswer, so the input contract did not move. RAG impact: no retrieval behaviour change - deterministic follow-up composition only; generation prompt untouched.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 53 minutes Limit details: You’ve used all 1 included review currently available under your plan. You completed 102 included PR reviews in the past 7 days; at that activity level, included reviews refill at 1 review per hour. Your organization has reached its usage spending cap. Adjust your spending cap in the billing tab. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (6)
Comment |
|
This pull request has been ignored for the connected project Preview Branches by Supabase. |
1 participant
Summary
src/lib/answer-follow-up.tsin place, closing Track A of the RAG improvement programme.buildAnswerFollowUpSuggestionsnow derives its candidates from the S2 related-information menu (buildRelatedInformationMenu,src/lib/rag/answer-composition.ts) for the answer's own query class and heuristic intent, instead of the previous fixed template list keyed off a regex over the question text.answerSectionsentry is skipped, and so is a question whose concept words already appear in the answer body. When nothing survives, the function returns an empty list and no chip row renders. That is the intended conservative outcome for a clinical surface, not a regression.none(document_lookup,unsupported_or_general) keep the previous narrow templates, now run through the same evidence gate and suppression, so §A2's "stays narrow" rule holds without deleting the surface for document lookups.RagAnswerfield, no new render block, no generation-prompt change, and no edit to anything undersrc/lib/rag/**— that directory is only imported from.ClinicalDashboard.tsxis unchanged because the input contract did not move:queryClassandqueryAnalysis.intentalready ride onRagAnswer.buildAnswerFollowUpQueryis untouched.tests/answer-follow-up.test.tscovers menu derivation per class and intent, the evidence gate with positive and discriminating negative cases, both suppression paths, and an alignment assertion over all 48 class×intent cells so a future S2 menu edit fails offline instead of silently dropping a chip.tests/answer-follow-up-chips.dom.test.tsxrenders the chip component with each call site's exact props — the desktop answer surface and the phone composer dock — and pins both call sites to the same function.docs/rag-improvement/HANDOVER.mdopens the S3 status row and corrects the stale S2 row (S2 merged 2026-08-18 as squashdda4956ff, canary pair 32100681177 -> 32111839806 green,eval:answer-qualityneutral with the owner's blinded read pending).docs/rag-behaviour/behaviour-map.md§8 records that S3 is a read-only consumer of the composition menu.RAG impact: no retrieval behaviour change — deterministic follow-up composition only; generation prompt untouched
Verification
npm run verify:pr-localHeavy/executable scope was selected as expected for a
src/lib/**change. Decisive lines:The unit suite's 8 failures are in 6 files this diff does not touch, and none of them import the changed module:
Proved rather than asserted.
tests/session-start-hook.test.tsfails identically at the merge base(
e1749bf8d, checked out in a separate detached worktree, never a stash):Test Files 1 failed | 4 passed (5).The other five pass when re-run in isolation on this branch —
Test Files 3 passed (3) / Tests 72 passed (72)and
Test Files 2 passed (2) / Tests 45 passed (45)— so those are Windows parallel-load timeouts in a7,116-test run, not regressions. The five gate steps the failing
teststep blocked were then run directly on thesame commit and all passed, as shown above.
Focused proof, run first and re-run after
npm run format:The S2 composition menu is unchanged by this PR, proved directly:
npm run verify:uiandnpm run verify:phone-chromewere deliberately skipped. No shared composer chrome, layout, reserve, or styling changed — the chip components, their props, their layouts, and the phone dock reserve wiring (hasAnswerFollowUps) are all byte-identical. The only change reaching the DOM is which strings the existing chip row receives, and that is proved in jsdom on both surfaces bytests/answer-follow-up-chips.dom.test.tsx.eval:answer-quality, noeval:rag, no Supabase read. This change is deterministic client-side composition over an answer that already exists; retrieval, ranking, selection, routing, and the generation prompt are all untouched.npm run check:production-readinesswas not run: answer generation, ingestion, source governance, privacy, and environment behaviour are unchanged by this diff.verify:pr-localalready raneval:rag:offlineandeval:rag:adversarial:offlinefor the RAG-adjacent scope.Risk and rollout
trimSourceForClient), so the gate can under-suggest but never over-suggest. Nothing about the answer, its citations, its verification, or its render trust ladder changes.Clinical Governance Preflight
Clinical KB Database(sjrfecxgysukkwxsowpy)The chips are clinical output: each one asserts that this corpus can answer that question next. The evidence gate is what makes that assertion honest, and the suppression rules stop the surface from offering back a question the answer already answered. No claim rendering, citation requirement, or degradation path changed, and the failure direction is conservative (fewer chips, never an unsupported one). Deployment classification is unaffected: this adds no new clinical decision-support behaviour — it narrows an existing suggestion surface.
Notes
Before / after for the ten requested questions
Generated offline from hand-built
RagAnswerfixtures — realistic query classes, intents, answer bodies, emitted sections, and short source snippets — by a scratchpad-only harness run twice: once at the merge base (e1749bf8d) before any edit, then again on this change. These are fixture outputs, not live answers; no provider call was made. The fixtures are not committed, so this table is illustrative evidence of the composition change, and the committed tests are the durable contract.medication_dose_risk/drug_dosing)medication_dose_risk/drug_dosing)medication_dose_risk/protocol)broad_summary/protocol)document_lookup/document_lookup)medication_dose_risk/drug_dosing)medication_dose_risk/drug_dosing)medication_dose_risk/escalation_risk)medication_dose_risk/escalation_risk)broad_summary/protocol)What the table shows, case by case:
escalation_riskintent — the escalation menu, which is S2's only intent refinement.monitoring_timingsection, and drops the renal chip because that fixture's clozapine snippet never mentions renal or hepatic dosing. Row 6 drops the renal chip because the answer itself gave the eGFR limits, and row 7 drops the contraindications chip because the answer emitted acontraindications_cautionssection. Before, all four questions received the identical four chips regardless of what the corpus held or what the answer had already said — including "What about pregnancy or breastfeeding?" on a metformin renal-dosing answer whose sources say nothing about pregnancy.none-menu path staying narrow: the monitoring/follow-up chip is suppressed because the answer already mentions follow-up, and the key-action-points chip is gated out because that fixture's snippet does not use those words.Follow-up
Track A of the programme closes here. Outstanding owner decisions are all in S7+: the Gate B verdict for the Docling lab, B5/B6, and the B7 dataset. Issue
#SDQSFD(ci-change-scoperag_eval_changedmissessrc/lib/rag/**) is queued for its own PR and deliberately untouched here.