test(eval): force-embedding flag + 10 vector-exercising golden cases by BigSimmo · Pull Request #249 · BigSimmo/Database

BigSimmo · 2026-07-03T13:01:04Z

Why

The golden retrieval set was 100% lexical fast-path (embedding_skipped_rate=1.0) — every case was answered by a lexical shortcut, so the eval could not detect whether a re-index changes vector/embedding retrieval quality. This unblocks re-index measurability (the RAG-optimization initiative's next gate).

What

forceEmbedding option on searchChunksWithTelemetry (SearchChunksArgs): bypasses every lexical text-fast-path so retrieval always exercises the embedding/vector stage. Diagnostic/eval-only, folded into the search cache key, never set on production paths (3 fast-path guards gated behind it).
eval-retrieval.ts: a per-case forceEmbedding field (optional) + a global --force-embedding flag.
10 vector-* golden cases (psychiatric monographs: PTSD, OCD, panic, anorexia, GAD, Tourette, postnatal, bipolar, ADHD, opioid) with forceEmbedding: true. Each is a clinical query that must be answered by vector retrieval of the correct monograph.

Why force embedding instead of wording

I first tried to craft queries that naturally fall through to the vector path. It's not reliably possible: the fast-path is driven by emergent lexical-match strength and substring title-rescue (hasDirectTitleSupport fires on any query word that is a substring of any top-4 retrieved title), which can't be controlled from wording. Forcing embedding is also the correct instrument — for a re-index you want to measure the vector index directly, not let a lexical shortcut mask a regression.

Verification

Live golden eval (eval:retrieval:quality): 34/34 pass (24 existing + 10 new), document_recall@5=1.0, content_recall@5=1.0; all 10 new cases run strategy=hybrid with embedding:1 (embedding actually used).
verify:cheap (runtime + lint + typecheck + vitest): green, 980 tests.
Existing 24 golden cases untouched (fixture diff is +194 insertions only).

🤖 Generated with Claude Code

The golden retrieval set was 100% lexical fast-path (embedding_skipped_rate=1.0), so it could not measure whether a re-index changes vector/embedding retrieval quality. - forceEmbedding option on searchChunksWithTelemetry (SearchChunksArgs): bypasses every lexical text-fast-path so retrieval always exercises the embedding/vector stage. Diagnostic/eval-only; folded into the search cache key; never set on production paths. - eval-retrieval.ts: per-case `forceEmbedding` field + a global `--force-embedding` flag. - 10 `vector-*` cases (psychiatric monographs: PTSD, OCD, panic, anorexia, GAD, Tourette, postnatal, bipolar, ADHD, opioid) with forceEmbedding=true. Each is a clinical query that must be answered by vector retrieval of the right monograph — verified live at document_recall@5=1.0, content_recall@5=1.0, all via strategy=hybrid (embedding used). Rationale: forcing embedding is the correct instrument for re-index measurement — you want to measure the vector index directly, not have a lexical shortcut mask a regression. Wording alone can't reliably force the vector path (the fast-path is driven by emergent lexical-match strength), so the flag makes these probes deterministic. Live golden eval: 34/34 pass (24 existing + 10 new), no regression. verify:cheap green (980). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4f03aeffe5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…te flag - Shared rag_response_cache: key retrievalPlanCacheQuery by forceEmbedding so a cached non-forced text-fast-path result can't satisfy a forced vector run (comment 3520042053). - Bypass the evidence-coverage-gate early return under forceEmbedding, so forced cases can't return before the embedding stage (comment 3520042055). - Isolate the vector index for forced cases: drop the pre-embedding lexical/memory/table candidates and run match_document_chunks_hybrid vector-only (empty query_text), so a broken vector index can't be masked by the lexical text-candidate path (comments 3520042061). - eval-quality.ts (eval:quality:release) now forwards testCase.forceEmbedding so the release gate honours the vector-* annotations instead of taking lexical shortcuts (comment 3520042059). - Prettier-format the fixture. Re-validated live under isolation: 34/34 golden pass, document_recall@5=1.0, content_recall@5=1.0, all 10 vector-* cases strategy=hybrid embedding:1. verify:cheap green (980). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3bdb703510

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-07-03T15:16:29Z

      const { data, error } = await supabase.rpc("match_document_chunks_hybrid", {
        query_embedding: embedding as unknown as string,
-        query_text: textSearchQuery,
+        query_text: args.forceEmbedding ? "" : textSearchQuery,


Make all forced RPCs vector-only

Fresh evidence after the prior isolation fix: this only blanks query_text for match_document_chunks_hybrid, while the same forced path still calls searchEmbeddingFieldCandidates and searchIndexUnitCandidates with buildClinicalTextSearchQuery(args.query). Those hybrid RPCs select/boost lexical text hits, so a vector-* golden case can still pass through embedding-field or index-unit text branches even if the chunk vector index/regeneration is broken; propagate the forced mode to those helpers or assert vector-only provenance before treating these cases as vector probes.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-07-03T15:16:29Z

    latencyMs: telemetry.embedding_latency_ms,
  });

+  if (args.forceEmbedding) {


Fail forced evals when embeddings fail

Because the lexical candidates are cleared only after embedTextWithTelemetry succeeds, the default RAG_PROVIDER_MODE=auto failure path above this block still returns the pre-embedding lexical textFastResults on quota/rate-limit/transient embedding errors. In that scenario a forced vector-* eval can pass without exercising any vector retrieval, which defeats the new guard; when forceEmbedding is set, the embedding failure should fail the search or at least not fall back to lexical results.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-07-03T15:16:29Z

        topK: testCase.topK,
        minSimilarity: 0.12,
        skipCache: true,
+        forceEmbedding: testCase.forceEmbedding,


Gate release eval on forced misses

Forwarding forceEmbedding here makes eval:quality:release run the new vector probes, but that command still exits based only on the aggregate 0.8 retrieval thresholds in buildEvalQualityReport; with 34 cases, six forced vector misses can still leave hit/recall at 28/34 = 0.823 and pass release. If these cases are meant to protect re-index quality, forced-case failures (or any golden failed_cases) need to be blocking rather than only reported.

Useful? React with 👍 / 👎.

* issues: clarify #250 waves do not outrank A1 acuity Orphaned follow-up from PR #1624 auto-merge race. Wave packaging is the engineering track only; recommended-queue A1 rows stay acuity-first, and Wave 0 queue repair is marked landed with the #249–#251 capture. Co-authored-by: BigSimmo <BigSimmo@users.noreply.github.com> * ledger: record prlanded for merged PR #1624 Squash b03b51d landed the 1:1 queue repair; note orphaned #250 clarify fix-forwarded on this branch. Co-authored-by: BigSimmo <BigSimmo@users.noreply.github.com> * fix(issues): clarify #250 A1 track vs engineering waves Separate approval-gated A1 work from Wave 4, keep #22/#183 as A2 operator track, and restore append-only ledger order for the #1624 prlanded row. Co-authored-by: BigSimmo <BigSimmo@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: BigSimmo <BigSimmo@users.noreply.github.com>

chatgpt-codex-connector Bot reviewed Jul 3, 2026

View reviewed changes

Comment thread src/lib/rag.ts Outdated

Comment thread src/lib/rag.ts

Comment thread scripts/fixtures/rag-retrieval-golden.json

Comment thread src/lib/rag.ts

BigSimmo enabled auto-merge (squash) July 3, 2026 14:59

BigSimmo and others added 2 commits July 3, 2026 23:02

Merge branch 'main' into claude/vector-golden-cases

3bdb703

BigSimmo merged commit f76842a into main Jul 3, 2026
4 checks passed

chatgpt-codex-connector Bot reviewed Jul 3, 2026

View reviewed changes

BigSimmo deleted the claude/vector-golden-cases branch July 4, 2026 06:23

BigSimmo mentioned this pull request Aug 12, 2026

chore(process): harden agent handoff, ledger revalidation, and wins classifier #1840

Merged

Navigation Menu

test(eval): force-embedding flag + 10 vector-exercising golden cases - #249

test(eval): force-embedding flag + 10 vector-exercising golden cases#249
BigSimmo merged 3 commits into
mainfrom
claude/vector-golden-cases

BigSimmo commented Jul 3, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Jul 3, 2026

Uh oh!

chatgpt-codex-connector Bot Jul 3, 2026

Uh oh!

chatgpt-codex-connector Bot Jul 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

BigSimmo commented Jul 3, 2026

Why

What

Why force embedding instead of wording

Verification

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant