feat(eval): add an offline answer-quality and retrieval eval harness

[felladrin]

Description

Closes #2192.

There was no eval harness, so a change to the prompt, the reranker, or the model shipped blind: nothing measured whether the ranking or the answer got better or worse.

This PR adds a small offline eval in eval/, driven by a fixed golden set of 26 queries. It is a regression baseline, not a benchmark.

Retrieval

Runs the real ONNX reranker through rankSearchResults (with preserveTopResults=true, the same call the app makes for text search) and scores the ranking with nDCG@3 and recall@3 against the labeled relevant results. It fails if the mean drops below a floor set in the middle of the band between a working reranker and a no-op (identity) reranker, with headroom on both sides: the golden set is curated so a no-op reranker scores 0.587 / 0.500 while a working one scores 0.950 / 0.981, and the floors (0.75 / 0.70) sit between them. A "stays falsifiable" test pins the no-op baseline below the floors, so a future edit to the set that breaks the premise (both relevant results in the input top-3) fails loudly. The headroom is intentional: the reranker is a quantized model whose scores drift across execution providers.

Answer

Builds the exact prompt the app sends (the real getFormattedSearchResults + getDefaultChatMessages, with pubSub mocked to the golden query), asks a chosen LLM backend for an answer, and grades it with a separate LLM judge against a rubric. It also checks a deterministic citation rate and a snippet-only-facts aggregate. It is gated on EVAL_LLM_API_KEY and skips cleanly without one.

What changed

Area	Change
eval/goldenSet.ts	26 hand-curated queries: labeled relevant results, reference answers, rubrics. Includes fictional snippet-only facts (the model cannot know them) and entries whose first candidate is irrelevant, so the pinned-first branch can cost the ranking.
eval/metrics.ts	Pure nDCG@k / recall@k (binary relevance).
eval/retrieval.integration.test.ts	Real reranker on the golden set; fails below the floor.
eval/answer.integration.test.ts	App prompt + LLM judge, plus citation-rate and snippet-only checks. Key-gated.
eval/goldenSet.test.ts, eval/promptConstruction.test.ts, eval/metrics.test.ts	Default suite: golden-set structure, prompt construction, metrics.
eval/goldenPrompt.ts	Shared prompt builder used by both prompt tests, so they cannot drift.
shared/defaultSystemPrompt.ts	The default system prompt, extracted so the client and the eval share one source (a stale copy would let prompt regressions ship green).
client/modules/settings.ts	Uses the shared prompt constant.
tsconfig.eval.json, vitest.eval.config.ts	Type-check and run the model/network tests separately (node env, no jsdom setup).
package.json	eval, eval:retrieval, eval:answer scripts; lint now type-checks the eval.
docs/development-commands.md, eval/README.md	Docs.

How to test

1. npm run lint (clean, including the new tsc -p tsconfig.eval.json).
2. npm test (the golden-set, prompt-construction, and metrics tests run here, so they are covered in CI).
3. npm run eval:retrieval (needs the model in server/models/; local only, CI has no model on disk). Baseline on my machine: mean nDCG@3 0.950, mean recall@3 0.981 (floors 0.75 / 0.70).
4. npm run eval:answer (needs EVAL_LLM_API_KEY; I could not run it here, no key).

Known limitations

• The answer eval's LLM-judged path is not exercised here (no API key). On a first keyed run it is worth confirming that deleting the citation paragraph from DEFAULT_SYSTEM_PROMPT drops the printed citation rate under 0.8 (the results block shows markdown links in-context, so the model may still cite).
• The prompt is built for the no-page-content case (pageContents = {}), so allocatePageExcerpts / formatExcerpt never appear in a graded prompt even though enablePageContentFetch defaults to true. "The exact prompt the app sends" holds for the no-page-content case.
• The answer eval hardcodes temperature: 0 (deliberate, for determinism) and max_tokens: 4096 rather than the app's getDefaultChatCompletionCreateParamsStreaming() (temperature: 0.35), so a change to the app's sampling params is invisible to it. max_tokens / temperature are also rejected by o-series / GPT-5-style endpoints (max_completion_tokens, fixed temperature); the documented default is gpt-4o-mini.
• Roughly a third of the rubric points are negative ("does not give a wrong year"), which LLM judges pass by default; that is a judge-variance source, so the citation rate and the snippet-only aggregate do the real work. The snippet-only floor (0.8) spans 3 entries x 2 rubric points = 6 binary decisions, so it is effectively "at least 5 of 6".
• No retry on transient LLM errors (a single 429 / 5xx fails the run); the answer eval is local-only and key-gated, so this is a false-red source rather than a correctness one.

Note: the golden-set validator rejects $& / $` / $' / $$ in result fields, which fences the eval off from the pre-existing String.replace bug in systemPrompt.ts. That bug stays separate and out of scope here.