Fix flowchart-next-step retrieval: surface red-zone action evidence - #119
Conversation
The golden case "In the clinical flowchart, what is the next step after red-zone risk?" failed content-recall because the answer chunks (red zone -> escalate / urgent senior review) never reached the top 5. Three compounding causes, smallest fix applied at each level: 1. Candidate generation (root cause): the flowchart query variants "red zone risk flow" and "risk flow review urgent escalation" are fully conjunctive under websearch_to_tsquery and matched 0 and 2 live chunks - they contributed nothing. Replaced with a plain "red zone" variant (13 precise zone-action chunks live) gated on the query mentioning a zone. 2. Fast-path gating: both decideTextFastPath (document_lookup) and the visual_flowchart_risk_gate accepted action-free flowchart pages - the gate satisfied its two term groups across different results and image captions. Flowchart/zone action queries now require zone AND action language on a single top result before skipping structured retrieval (mirrors threshold_action_requires_structured_retrieval). 3. Ranking: riskFlowchartSource only recognised flowchart-worded text, so escalation protocols expressing the flowchart steps as prose took the -0.18 generic penalty while unrelated risk-assessment flowcharts kept the +0.16 boost. Zone+action text now counts as risk-flowchart source evidence. Validation: golden retrieval eval 10/10 (content_recall@5 1.0, up from 0.9667; doc_recall@5 1.0; hit_rate 1.0), latency eval 0 failures (median 1.2s, p90 12.1s), fixed case answers in ~1.0s via fast path with the correct evidence ranked #1. Full vitest 691 passed, typecheck and lint clean. Three new unit tests pin the gates and ranking rule. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3a6f02b106
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
- Fast-path guard now requires zone context AND action language on a single top result (was: any action word in top-5), matching the coverage gate, so "review <anything>" on an unrelated flowchart can no longer short-circuit structured retrieval. - The injected zone variant follows the colour the query names (was: always "red zone" for any zone mention), so amber/yellow-zone questions no longer pull red-zone chunks into their candidate pool. - riskFlowchartZoneActionSource requires an explicit coloured-zone reference or "zone criteria" (was: bare "zones?"), so generic zone+review chunks keep the intended penalty. - All three sites now share riskZoneContextPattern / riskZoneActionPattern exported from clinical-search to prevent drift. Validation: golden eval 23/23 (all metrics 1.0, flowchart case 1.1s), full vitest 741 passed, typecheck + lint clean. Two new test cases pin the action-without-zone refusal and the colour-matched variant. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3fc2a84869
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…ck-9f3854 # Conflicts: # src/lib/clinical-search.ts # src/lib/rag.ts
There was a problem hiding this comment.
Pull request overview
This PR fixes a failing “flowchart-next-step” retrieval scenario by tightening candidate generation, fast-path gating, and ranking so that coloured-zone “next step” questions surface zone-action escalation evidence (rather than action-free flowchart pages).
Changes:
- Adjust retrieval query variants for flowchart/zone next-step queries by replacing overly-conjunctive variants with a
<colour> zonevariant (matched to the colour named in the query). - Strengthen the document-lookup fast path and evidence coverage gate so zone-action queries require zone + action language on a single top result before skipping structured/vector retrieval.
- Update ranking signals so zone-action escalation prose counts as risk-flowchart evidence even without the literal word “flowchart”, and add unit tests to pin the behaviors.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| tests/retrieval-query-variants.test.ts | Adds unit tests covering the new zone-action gating behavior, zone-colour variant generation, and ranking treatment of zone-action prose. |
| src/lib/rag.ts | Updates variant generation and fast-path/coverage gating to require zone-action evidence using shared patterns. |
| src/lib/clinical-search.ts | Introduces shared zone context/action regex signals and expands risk-flowchart source recognition for ranking. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 21f65b9a68
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
- isRiskFlowchartNextStepQuery accepts any coloured-zone query (amber/ yellow/etc), not just risk|red, keeping the guard aligned with the multi-colour variant handling (Copilot). - hasRiskFlowchartActionEvidence now matches the colour the query names (a red-zone question cannot fast-path on an amber-zone chunk) and accepts bare colour tokens on risk-matrix/flowchart visual units, whose content stores the cell colour as "... | Red | escalate ..." (Codex P2 x2). - The coverage-gate zone branch now only catches zone/next-step queries; plain flowchart document lookups fall through to the ordinary title gate instead of being rejected for lacking zone evidence (Codex P2). - Ranking source check mirrors the risk-matrix colour-token rule. Validation: golden eval 23/23 (flowchart case 1.3s), full vitest 780 passed, typecheck + lint clean. Four new test cases pin the colour matching, matrix-cell tokens, and title-gate fallthrough. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…into claude/sweet-mcclintock-9f3854 # Conflicts: # src/lib/rag.ts
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
…into claude/sweet-mcclintock-9f3854 # Conflicts: # src/lib/rag.ts # tests/retrieval-query-variants.test.ts
…into claude/sweet-mcclintock-9f3854 # Conflicts: # src/lib/rag.ts
isRiskFlowchartNextStepQuery now matches "risk-matrix" like the variant block already does, so hyphenated phrasings get the same action-evidence guard and coverage gate. Test pins the hyphenated form. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…into claude/sweet-mcclintock-9f3854
There was a problem hiding this comment.
💡 Codex Review
Lines 3510 to 3511 in 55b424d
Apply the zone-action gate to risk-matrix queries
Risk-matrix next-step/action queries do not reach this branch in the normal path: phrases like What action is shown for the risk matrix red zone? are classified as table_threshold because matrix matches the table-threshold signals, while this guard only runs inside document_lookup. In that scenario the table-threshold fast path only checks for structured evidence and score, so an off-colour risk_matrix_cell can still short-circuit without the colour/action guard that this change adds for risk-matrix cells; classify these risk-matrix action lookups into this path or apply the same guard before the table-threshold fast path accepts them.
Database/src/lib/clinical-search.ts
Lines 1412 to 1414 in 55b424d
Gate canonical flowchart titles on action evidence
Fresh evidence for the previously reported action-free boost class: this new nextStepActionQuery guard only wraps riskFlowchartSource, but the riskFlowchartCanonicalBoost path immediately below still gives +0.32 to a title like Risk Assessment Flow Chart even when the result has no escalation/action text. For red-zone next-step queries that page still nets a positive risk-flowchart boost after the generic penalty and can rank ahead of the actual zone-action prose, so the canonical-title boost should also require same-result action evidence for next-step/action queries.
Database/src/lib/clinical-search.ts
Line 59 in 55b424d
Keep zone criteria colour-scoped
Fresh evidence for the colour-scoping regression: even when zoneContextPatternsForQuery() narrows colourGroup to the requested colour, this alternative still lets any result containing zone criteria satisfy the zone side of the guard. A red-zone next-step query can therefore fast-path or receive the zone-action ranking boost from an Amber Zone criteria ... urgent senior review chunk, so the generic zone criteria fallback needs to be disabled or colour-qualified when the query names a specific colour.
Line 1963 in 55b424d
Add action/after zone queries to variant injection
The new colour-zone retrieval variant is skipped for flowchart queries phrased with action/after but without risk, urgent, escalat, or the exact words next step, e.g. In the clinical flowchart, what action is shown after amber zone?. Those queries still trigger the zone-action guard later, but the precise amber zone text RPC variant is never added, leaving only the fully-conjunctive primary query to recall the action chunk; include the same action/after triggers used by isRiskFlowchartNextStepQuery.
Database/src/lib/clinical-search.ts
Lines 1412 to 1414 in 55b424d
Colour-scope the lexical risk-flowchart fallback
Fresh evidence for the earlier colour-scoped ranking issue: the next-step branch can still accept riskFlowchartLexicalSource && riskZoneActionPattern without checking the requested zone colour. For a red-zone question, a result like Risk flowchart: Amber Zone, escalate for urgent review has flowchart + risk + action, so it gets the risk-flowchart boost and avoids the generic penalty even though riskFlowchartZoneActionSource correctly rejected the amber evidence; make this fallback use the colour-scoped zone context too.
Lines 3288 to 3290 in 55b424d
Guard zone next-step queries without flowchart wording
Natural zone-action questions such as What action is required after red-zone criteria? do not satisfy this predicate unless the user also says flowchart/algorithm/pathway/risk-matrix. Those queries are otherwise classified through the threshold path because criteria is a threshold signal, where a high-scoring criteria-only chunk can fast-path without the action evidence this guard is meant to require; apply the same zone-action guard whenever the query names a zone and asks for the next action.
Database/src/lib/clinical-search.ts
Lines 1379 to 1380 in 55b424d
Recognize non-red zone flowchart sources
By adding riskZoneContextPattern to riskFlowchartQuery, document lookups like Where is the Amber Zone flowchart? now enter the risk-flowchart ranking branch, but the source test still only recognizes risk/red unless the result also carries action language. A correct Amber Zone Flowchart title/content with no action instruction therefore gets the generic risk-flowchart penalty, so the lexical source check should accept the queried colour rather than hard-coding red/risk.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Whole-table conflict in docs/outstanding-issues.md — main advanced 44 commits and gained rows while this PR sat open. Resolved by rebuilding from main's current file and re-applying this branch's three changes, not by taking either side: settling a whole-table conflict by picking a side is the row-loss failure ledger #112 exists to make loud. Renumbered the new row #117 -> #125. When this branch opened it, the marker was 117; main's marker is now 125 and #117..#119 were reserved by other sessions in the meantime, so keeping #117 would have collided with a row someone else is about to add under that number. Ids are never reused — renumber the incoming row above the marker and bump it. Audited row-by-row against origin/main rather than eyeballed: main rows: 119 resolved rows: 120 LOST from main: (none) ADDED vs main: #125 MOVED table: #113 open->archive, #114 open->archive duplicates: (none) check:outstanding-issues: 120 rows (58 open, 62 archived), unique ids, next-id=126 above the highest. verify:cheap exit 0, Tests 4493 passed | 4 skipped. format:check clean. `c2edda18` still absent from the file. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FvU8z73P6TXUXoYBqN5K1P
Main's side carried #128-#130, archived #116 and reordered #117-#119, so it is taken whole; only the #127 row is grafted from this branch, since main still holds the withdrawn "sharedChromePinned is stuck" text. Marker stays at main's 131. The file now reports `union merge active`, so this class of collision should stop. Also cross-links #130 into #127. PR #1396 ("overlay the phone header so hiding it never moves content") merged at 06:49:55, about twenty minutes before the first failure, and rewrote the exact overlay path this defect sits in — phoneOverlayMotion, the overlay stack transition, and readChromeCollapseMetrics charging zero released top geometry. Its merge commit 90b3e34 is the single main run that passed, which fits the passed-once-then-failed-twice-escalating shape better than the focus-latch reading. #130 also records that #1396's own declared physical-device prerequisite was never run, and that headless Chromium cannot certify what it covers. Verified after the merge: check:outstanding-issues 128 rows / unique ids / next-id=131 / union merge active; check:branch-review-ledger 97 live + 1206 archived; prettier clean on all three changed files. Not re-run after the merge: verify:cheap and the phone-scroll spec, which both passed on this branch's own commit before it (434 test files / 4562 tests, and 56 passed). The merge adds only main's already-verified tree plus one docs cell, so the earlier evidence still covers this diff — but it is the earlier commit's evidence, not this one's. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XrPbbfU9yWuEjEVypCr4ZQ
Fifth consecutive genuine conflict on docs/outstanding-issues.md, confirmed with git merge-tree before acting. Main's d8b034c (PR #1441) deleted three rows from the Open items table and archived none of them: #85, which that PR genuinely resolved by adding check-upload-limit-parity.mjs; #119, a duplicate of #122; and #105, which is open and unresolved — its LoadingPanel half is still unverified. This resolution mirrors main on #85 and #119, which were dispositionable, and keeps #105, which was not. #105 survived only because this resolution was done row by row and then diffed with column padding normalised away. Taking either side wholesale would have lost it silently, which is what AGENTS.md forbids for this file and what appears to have happened upstream. Filed as #148: check:outstanding-issues validates duplicate ids, ids in both tables, stale markers and malformed rows, but never compares against the previous revision, so a row that simply disappears is invisible to it in verify:cheap and static-pr alike. The fix is a deletion check against the merge base, not a union merge driver — #133 removed that driver deliberately and reinstating it would bring back the duplicate-row damage it was removed to stop. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01361jh3eYVjJCzXWjAhdZiF
Summary
Fixes the failing
flowchart-next-stepgolden retrieval case (content_recall@5 0.67 → 1.0) with the smallest fix at each of three compounding layers:"red zone risk flow"/"risk flow review urgent escalation"are fully conjunctive underwebsearch_to_tsqueryand matched 0 / 2 live chunks — dead variants. Replaced with a plain"red zone"variant (13 precise zone-action chunks live), gated on the query mentioning a zone.decideTextFastPathandvisual_flowchart_risk_gateaccepted action-free flowchart pages (term groups satisfied across different results / image captions). Zone-action queries now require zone AND action language on a single top result before skipping structured retrieval — mirrors the existingthreshold_action_requires_structured_retrievalpattern.riskFlowchartSourceonly recognised flowchart-worded text, so escalation protocols (the actual answers) took the −0.18 generic penalty while unrelated risk-assessment flowcharts kept the +0.16 boost. Zone+action prose now counts as risk-flowchart evidence.Top result for the query is now: "If deteriorating (has any Purple or Red Zone criteria on observation chart), escalate for Senior Clinician Review or call a MET" (Sepsis Pathway Adult Guideline), answered in ~1.0s via the fast path.
Clinical governance preflight
Touches search/ranking. Golden retrieval eval 10/10 (content_recall@5 1.0 ↑ from 0.9667, doc_recall@5 1.0, hit_rate 1.0, mrr 0.745); latency eval 0 failures (median 1.2s, p90 12.1s); full vitest 691 passed; typecheck + lint clean. Three new unit tests pin the gate and ranking behavior.
🤖 Generated with Claude Code