Optimize retrieval search and simplify document search home - #85
Conversation
There was a problem hiding this comment.
Pull request overview
This PR improves retrieval performance and reliability by increasing pgvector HNSW recall, reducing redundant/serial Supabase RPC work during RAG retrieval, and tightening worker/agent completion semantics, while also simplifying the Documents search “home” UI state.
Changes:
- Raise pgvector
hnsw.ef_searchfor key retrieval RPCs to better match their deeper candidate fetch limits. - Speed up retrieval by memoizing per-request memory-card fetches and running independent embedding-based RPCs concurrently.
- Harden indexing completion by introducing strict enrichment completion via RPC and simplifying the document search empty state UI.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| worker/main.ts | Adds strict enrichment completion RPC flow; reuses image bytes for caption+upload; refactors metadata/agent repair flags. |
| tests/worker-visual-capture.test.ts | Updates worker source expectations for new strict completion and agent-repair logic. |
| tests/ui-smoke.spec.ts | Adjusts UI smoke assertions for simplified Documents empty state and tag/button expectations. |
| tests/indexing-v3-agent.test.ts | Adds coverage asserting the agent recognizes local-worker visual units as satisfying artifact capture. |
| tests/document-index-units.test.ts | Expects generated_by: "local-worker" in visual index unit metadata. |
| supabase/schema.sql | Records per-function hnsw.ef_search settings for retrieval functions. |
| supabase/migrations/20260627000000_retrieval_hnsw_ef_search.sql | Introduces migration to set hnsw.ef_search = 100 on retrieval functions. |
| supabase/functions/indexing-v3-agent/index.ts | Broadens “visual unit generated” detection to include local-worker / visual-intelligence metadata. |
| src/lib/rag.ts | Adds per-request memory-card memoization and parallelizes independent embedding-based retrieval RPCs. |
| src/lib/document-index-units.ts | Marks visual units as generated_by: "local-worker". |
| src/components/ClinicalDashboard.tsx | Removes unused document-search callbacks from DocumentSearchResultsPanel usage. |
| src/components/clinical-dashboard/document-search-results.tsx | Simplifies the document search home/empty state UI and removes suggested-search wiring. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| const preparedImages = await Promise.all( | ||
| extracted.images.map(async (image) => { | ||
| const bytes = await readFile(image.path); | ||
| const imageHash = hashBytes(bytes); | ||
| return { | ||
| // B4: retain the buffer so caption (base64) and storage upload reuse this single | ||
| // read instead of re-reading the same file from disk two more times per image. | ||
| bytes, | ||
| imageHash, | ||
| bytesLength: bytes.length, |
| const effectiveMatchCount = Math.max(args.matchCount, 48); | ||
| const cacheKey = `${args.query}\0${args.queryEmbedding?.length ? "vec" : "text"}\0${effectiveMatchCount}`; | ||
| let cardsPromise = args.cardCache?.get(cacheKey); |
| enrichment_error: strictCompletionMessage, | ||
| indexing_v3_agent_status: "pending", | ||
| indexing_v3_agent_last_error: strictCompletionMessage, | ||
| indexing_v3_agent_repair_reason: "strict_completion_gate_blocked", |
| const preparedImages = await Promise.all( | ||
| extracted.images.map(async (image) => { | ||
| const bytes = await readFile(image.path); | ||
| const imageHash = hashBytes(bytes); | ||
| return { | ||
| // B4: retain the buffer so caption (base64) and storage upload reuse this single | ||
| // read instead of re-reading the same file from disk two more times per image. | ||
| bytes, |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c202e26eb8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| it("documents that local worker visual units satisfy visual artifact capture", () => { | ||
| const edgeSource = String( | ||
| await import("node:fs/promises").then((fs) => |
There was a problem hiding this comment.
When this spec is loaded, the new callback is a normal function but contains await import(...), which is invalid syntax inside a non-async function. Any vitest/npm run test run that parses this file will abort before executing tests; make the callback async or use a synchronous file read.
Useful? React with 👍 / 👎.
| ...(agentRepairRequired | ||
| ? { | ||
| indexing_v3_agent_status: "pending", | ||
| indexing_v3_agent_last_error: enrichmentErrorMessage ?? optionalRepairMessage, |
There was a problem hiding this comment.
With the default WORKER_INLINE_ENRICHMENT=false, agentRepairRequired is true because enrichment is deferred, but enrichmentErrorMessage is null and this line writes the optional-write failure message into indexing_v3_agent_last_error. That makes normal successful core indexing look like optional artifacts failed until the agent claims it; set a deferred-enrichment message/null unless optionalRepairRequired is actually true.
Useful? React with 👍 / 👎.
| return { | ||
| // B4: retain the buffer so caption (base64) and storage upload reuse this single | ||
| // read instead of re-reading the same file from disk two more times per image. | ||
| bytes, |
There was a problem hiding this comment.
For image-heavy PDFs/DOCX and default worker concurrency, this new bytes field keeps all extracted image buffers alive after the initial hashing pass, including images later skipped by cheap filters or caption budgets. That can push ingestion workers into high heap/RSS or OOM on large source documents; keep only hash/size/perceptual metadata here and reread or lazily cache bytes for the image currently being captioned/uploaded.
Useful? React with 👍 / 👎.
Fifth consecutive genuine conflict on docs/outstanding-issues.md, confirmed with git merge-tree before acting. Main's d8b034c (PR #1441) deleted three rows from the Open items table and archived none of them: #85, which that PR genuinely resolved by adding check-upload-limit-parity.mjs; #119, a duplicate of #122; and #105, which is open and unresolved — its LoadingPanel half is still unverified. This resolution mirrors main on #85 and #119, which were dispositionable, and keeps #105, which was not. #105 survived only because this resolution was done row by row and then diffed with column padding normalised away. Taking either side wholesale would have lost it silently, which is what AGENTS.md forbids for this file and what appears to have happened upstream. Filed as #148: check:outstanding-issues validates duplicate ids, ids in both tables, stale markers and malformed rows, but never compares against the previous revision, so a row that simply disappears is invisible to it in verify:cheap and static-pr alike. The fix is a deletion check against the merge base, not a union merge driver — #133 removed that driver deliberately and reinstating it would bring back the duplicate-row damage it was removed to stop. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01361jh3eYVjJCzXWjAhdZiF
Summary
Verification
npm run verify:cheapnpm run verify:uiwhen UI, routing, styling, browser behavior, reduced-motion, or forced-colors behavior changednpm run verify:releasebefore release or handoff confidence claimsnpm run format:checknpm run check:production-readinesswhen clinical workflow, privacy, environment, Supabase, source governance, or deployment behavior changedClinical Governance Preflight
Complete this section when the change touches ingestion, answer generation, search/ranking, source rendering, document access, privacy, production env, or clinical output.
Clinical KB Database(sjrfecxgysukkwxsowpy)Notes