Skip to content

Measure search quality on public benchmarks; fix BM25 word splitting - #2

Merged
YsK-dev merged 4 commits into
mainfrom
retrieval-eval
Nov 14, 2026
Merged

YsK-dev merged 4 commits into
mainfrom
retrieval-eval

Conversation

@YsK-dev

@YsK-dev YsK-dev commented Nov 14, 2026

Copy link
Copy Markdown
Owner

What changes

  • evaluation/retrieval_eval.py runs the app's own search engine on BEIR SciFact (5,183 abstracts, 300 test claims). It compares vector search, BM25, vector + reranker, the full hybrid + reranker pipeline, and the app's default call. Datasets and models are pinned.
  • BM25 fix: the tokenizer split text on spaces, so punctuation stayed on words. It now splits text into words. BM25 alone on SciFact: nDCG@10 0.560 → 0.652.
  • Independent check: on FiQA, which was not used to choose the fix, BM25 alone: 0.159 → 0.217. The full pipeline does not change measurably (0.694 → 0.696).
  • evaluation/compare.py: paired 95% bootstrap intervals.
  • evaluation/bm25_tokens_check.py: compares the old and new tokenizer on any BEIR set.
  • README section "How well the search works":
    • Measured results:
      • The reranker adds +0.046 nDCG@10 (95% interval +0.016 to +0.075).
      • BM25 on top of the reranker adds +0.006 (−0.009 to +0.022).
    • The app's default call: Hit@3 67.0%; 12% of claims get no passage. Without the cut-off, Hit@3 is 72.7%.
    • Limits of the test.

Checks

  • 35 tests pass with the full requirements.
  • With requirements-dev.txt only: 32 pass and 1 is skipped (the BM25 test needs the full requirements).
  • Every number in the README was recomputed from evaluation/results.json and the per-question CSV.

evaluation/retrieval_eval.py stores each SciFact abstract as one passage in the app's own
engine and runs the 300 test questions through vector search, BM25, vector + reranker, the
full hybrid + reranker pipeline, and the app's default call (3 passages, confidence cut-off 0.3).
Datasets and models are pinned. Results in evaluation/results.json, per question in
evaluation/per_question_scifact.csv. Metric checks in tests/test_retrieval_metrics.py.
Splitting on spaces kept punctuation on words, so "cancer." did not match "cancer".
On SciFact this raises BM25 alone from nDCG@10 0.560 to 0.652. The full hybrid + reranker
pipeline moves from 0.694 to 0.696 (no measurable change). Results re-run with the fix.
…rections

- evaluation/compare.py: paired bootstrap interval between two set-ups or two runs.
- evaluation/bm25_tokens_check.py: BM25 with the old and new word splitting on another BEIR
  dataset. On FiQA (not used to choose the fix): nDCG@10 0.159 -> 0.217, Hit@3 23.0% -> 30.9%.
- requirements-eval.txt: pandas and pyarrow for the measurement.
- README: the 10-result pipeline row is labelled as such; the default call's cost of the
  cut-off in full (17 claims, 156 with fewer than 3 passages); claims, not questions; whole
  abstracts, not the app's passages; history is the last 6 messages; tables split every 50
  rows; FAISS needs faiss-cpu; first start downloads two models.
@YsK-dev
YsK-dev merged commit 16a6463 into main Nov 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant