[4shut0sh]

…val CLI

Add a full retrieval evaluation framework so search quality can be measured, compared, and tracked instead of evaluated by vibes.

• src/core/search/eval.ts: pure IR metrics (precisionAtK, recallAtK, mrr, ndcgAtK), EvalQrel/EvalConfig/EvalReport types, and runEval() orchestrator that executes any strategy against user-provided qrels
• src/commands/eval.ts: gbrain eval CLI with single-run table and A/B comparison mode (--config-a / --config-b) showing Δ nDCG
• src/core/operations.ts: eval_search operation available via MCP; agents can pass inline qrels JSON and get metrics back
• src/core/search/hybrid.ts: add rrfK and dedupOpts to HybridSearchOpts so A/B tests can override RRF_K=60, cosine threshold, type ratio cap without touching constants
• test/eval.test.ts: 30+ unit tests for all metric functions
• test/e2e/mechanical.test.ts: replace non-asserting console.log precision@5 block with real expect() assertions using precisionAtK

Usage: gbrain eval --qrels ./my-queries.json
gbrain eval --qrels ./qrels.json --config-a '{"rrf_k":60}' --config-b '{"rrf_k":30}' Made-with: Cursor

[irresi]

#36

[garrytan]

@4shut0sh — this is exactly what GBrain needed. A proper retrieval eval framework with real IR metrics instead of vibing on search quality.

The API design is clean: pure metric functions with zero deps for unit testing, runEval() orchestrator for E2E, A/B comparison mode for parameter tuning. Making rrfK and dedupOpts configurable on HybridSearchOpts was the right call — it integrates perfectly with our compiled truth boost work.

We've pulled this into garrytan/search-quality-boost (PR #64) with full attribution via Co-Authored-By. It's now part of the search quality suite alongside the compiled truth ranking, cosine re-scoring, and source-aware dedup. All 38 eval unit tests pass.

Massive thank you for this. The eval harness is the kind of infrastructure that makes every future search improvement measurable instead of guesswork.

Closing in favor of #64 which includes this work.