Skip to content

Better embedding model, measured; cut-off and reranker studies; index records its model - #3

Merged
YsK-dev merged 7 commits into
mainfrom
better-retrieval
Nov 15, 2026
Merged

YsK-dev merged 7 commits into
mainfrom
better-retrieval

Conversation

@YsK-dev

@YsK-dev YsK-dev commented Nov 15, 2026 •

Copy link
Copy Markdown
Owner

What changes

  • Embedding model: bge-small-en-v1.5 replaces all-MiniLM-L6-v2. Three small models were compared with vector search alone (evaluation/embedder_choice.py). The rule, fixed in advance, was the best nDCG@10 on SciFact's 809 training claims:
    • training claims: bge-small 0.747, e5-small-v2 0.708, MiniLM 0.660;
    • SciFact test claims: +0.068 (95% interval +0.041 to +0.094);
    • FiQA, which played no part in the choice: +0.035 (+0.017 to +0.053).
    • Part of the SciFact gain comes from length: the old model reads only the first 256 tokens. When the answer fits in 256 tokens, the gain is +0.030 (−0.016 to +0.076); when it is longer, +0.083. The app's short passages should see a gain close to FiQA's.
  • Cost of the new model: turning documents into vectors takes 3 to 5 times longer, and searching takes 18 ms instead of 10 ms.
  • What the app returns did not change: its default call finds a right abstract in the top 3 for 67.0% of the test claims with either model, because the reranker picks the final 3.
  • Each index records its model, in the Chroma collection metadata or in faiss_meta.json:
    • an index built with another model is refused, before any model is loaded;
    • FAISS also checks the vector size and the count;
    • RAG_EMBEDDER accepts an empty value or a repo id;
    • at start-up, the app warns when the new index is empty but uploaded files exist.
  • Reranker: it is now loaded with raw scores (load_reranker, activation_fn=Identity) in the engine and in the evaluation scripts, so the sigmoid can never be applied twice. Numbers for the current reranker are unchanged, because its config already sets Identity.
  • New studies, all in the README:
    • Confidence cut-off (cutoff_tuning.py): the trade-off between answerable and unanswerable claims. The rule set in advance picked 0.95, but the rule favours refusing everything, so its choice was not adopted and 0.3 stays.
    • Vector search alone (dense_cutoff.py): on training and test claims, with and without a cut-off.
    • Larger rerankers (reranker_check.py): no candidate met the rule fixed in advance, so ms-marco-MiniLM-L-6-v2 stays.
  • Tests:
    • tests/test_embedders.py adds 14 tests that use a fake model with the real ChromaDB and FAISS;
    • a new CI job, engine-tests, runs them;
    • 51 tests pass locally, with and without RAG_TESTING=1.
    • CI could not run on this PR: GitHub did not start the jobs because the account is locked over billing.

Not changed, stated in the README

  • The reranker stays the default. With the new model the evidence on SciFact is mixed:
    • training claims: vector search alone is better, +3.1 points of Hit@3 (+0.5 to +5.6);
    • test claims: the two are level.
    • Testing whether it can be dropped on a second dataset is the next step.
  • Known, not part of this PR: an upload is embedded once per back end, FAISS re-embeds the corpus when a document is deleted, and uploads run in the request thread.

…rch without the reranker

- rag_engine.py: pinned embedders with their query prefixes; RAG_EMBEDDER chooses one; a new
  embedder gets its own index. bge-small-en-v1.5 chosen on the SciFact training claims.
- retrieval_eval.py: --embedder; results keyed by data set and embedder.
- cutoff_tuning.py: reranker confidence on an answerable and an unanswerable index, train and test.
- dense_cutoff.py: vector search with a similarity cut-off against the reranker path.
- reranker_check.py: larger rerankers, chosen on training claims (results to follow).
Tests: 37 passed.
…claims; check on FiQA with a rule fixed before running
…ient RAG_EMBEDDER; raw reranker scores everywhere; warn when the new index is empty; engine tests with a fake model, run in CI
…out cut-offs (from the saved per-claim results)
…ME: model table, costs, reranker section with train and test, caveats from review
… out of modify(); FAISS mismatch logged as such; sentence-transformers>=4.0; README: gain by text length, order of events, FiQA in MTEB
@YsK-dev
YsK-dev merged commit 049160d into main Nov 15, 2026
0 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant