Surfaced by a role-based review; verified against current code (no eval/benchmark exists anywhere).
Problem: There's no eval harness, golden set, retrieval-relevance metric, or citation-accuracy check. Every change to prompts, the reranker, or model defaults ships blind — no way to know if a change helped or hurt answers.
Fix: Add a small offline eval (eval/): a fixed set of ~30-50 queries with (a) labeled relevant URLs to measure reranker nDCG/recall, and (b) reference answers or a rubric scored by an LLM-judge against a chosen backend. Gives regression signal for prompt/reranker/model changes.
Files: new eval/ harness; exercises server/rankSearchResults.ts, client/modules/systemPrompt.ts, client/modules/textGenerationUtilities.ts
Surfaced by a role-based review; verified against current code (no eval/benchmark exists anywhere).
Problem: There's no eval harness, golden set, retrieval-relevance metric, or citation-accuracy check. Every change to prompts, the reranker, or model defaults ships blind — no way to know if a change helped or hurt answers.
Fix: Add a small offline eval (
eval/): a fixed set of ~30-50 queries with (a) labeled relevant URLs to measure reranker nDCG/recall, and (b) reference answers or a rubric scored by an LLM-judge against a chosen backend. Gives regression signal for prompt/reranker/model changes.Files: new
eval/harness; exercisesserver/rankSearchResults.ts,client/modules/systemPrompt.ts,client/modules/textGenerationUtilities.ts