Repository navigation
Better embedding model, measured; cut-off and reranker studies; index records its model - #3
Merged
Merged
Conversation
…rch without the reranker - rag_engine.py: pinned embedders with their query prefixes; RAG_EMBEDDER chooses one; a new embedder gets its own index. bge-small-en-v1.5 chosen on the SciFact training claims. - retrieval_eval.py: --embedder; results keyed by data set and embedder. - cutoff_tuning.py: reranker confidence on an answerable and an unanswerable index, train and test. - dense_cutoff.py: vector search with a similarity cut-off against the reranker path. - reranker_check.py: larger rerankers, chosen on training claims (results to follow). Tests: 37 passed.
…ff trade-off, rerankers tried
…claims; check on FiQA with a rule fixed before running
…ient RAG_EMBEDDER; raw reranker scores everywhere; warn when the new index is empty; engine tests with a fake model, run in CI
…out cut-offs (from the saved per-claim results)
…ME: model table, costs, reranker section with train and test, caveats from review
… out of modify(); FAISS mismatch logged as such; sentence-transformers>=4.0; README: gain by text length, order of events, FiQA in MTEB
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes
bge-small-en-v1.5replacesall-MiniLM-L6-v2. Three small models were compared with vector search alone (evaluation/embedder_choice.py). The rule, fixed in advance, was the best nDCG@10 on SciFact's 809 training claims:faiss_meta.json:RAG_EMBEDDERaccepts an empty value or a repo id;load_reranker,activation_fn=Identity) in the engine and in the evaluation scripts, so the sigmoid can never be applied twice. Numbers for the current reranker are unchanged, because its config already sets Identity.cutoff_tuning.py): the trade-off between answerable and unanswerable claims. The rule set in advance picked 0.95, but the rule favours refusing everything, so its choice was not adopted and 0.3 stays.dense_cutoff.py): on training and test claims, with and without a cut-off.reranker_check.py): no candidate met the rule fixed in advance, soms-marco-MiniLM-L-6-v2stays.tests/test_embedders.pyadds 14 tests that use a fake model with the real ChromaDB and FAISS;engine-tests, runs them;RAG_TESTING=1.Not changed, stated in the README