Ask questions about your own documents and get answers with their sources. Everything runs on your own machine: the documents, the search index and the language model (through Ollama).
This repository holds the back end, a FastAPI service. You can use it from the built-in API page
(/docs) or from any front end. The web front end shown in the design notes is not part of this
repository.
- Reads many file types: PDF (with tables), Word, Excel, CSV, PowerPoint, HTML, Markdown and plain text.
- Splits documents into passages: text is cut into pieces of about 500 characters with 100 characters of overlap, and small neighbouring pieces are joined up to 1,000 characters. Tables are kept as tables, 50 rows per passage with the header repeated. A passage that repeats, such as a page header, is kept at most twice.
- Finds the right passages in two steps:
- Candidates come from two searches: by meaning (vector search with
bge-small-en-v1.5in ChromaDB; FAISS also works afterpip install faiss-cpu) and by keywords (BM25). - A cross-encoder (
ms-marco-MiniLM-L-6-v2) reads each question–passage pair and puts the candidates in order. Passages it is not confident about (below 0.3) are dropped. If nothing is left, the app says it cannot answer instead of guessing.
- Candidates come from two searches: by meaning (vector search with
- Answers with a local model: Qwen3 1.7B (fast) or Qwen3 4B Thinking, through Ollama. The answer streams word by word and lists the passages it used.
- Basic safety checks: common prompt-injection phrases in questions are rejected.
- Conversations: follow-up questions see the last 6 messages (3 questions and answers).
Measured on SciFact, a public benchmark from
BEIR: 5,183 scientific abstracts and 300 test claims (short scientific statements), each with the
abstracts that support or contradict it. Every abstract is stored whole as one passage in the
app's own search engine (src/rag_engine.py); reading files and splitting them into the app's
smaller passages is not part of this test. Laptop CPU (Apple M1), no GPU.
Choices below were made on SciFact's 809 training claims and then checked once on the 300 test claims. That check is not fully independent: 196 of the test claims have an answering abstract that also answers a training claim, and SciFact's test claims are part of the public MTEB benchmark on which embedding models are usually compared. So the main choice was also checked on a second dataset, FiQA, that played no part in it. FiQA is in MTEB too, so it checks this project's choice, not the public rankings.
| Search set-up (10 results) | nDCG@10 | Recall@10 | Hit@3 | Time per claim |
|---|---|---|---|---|
| Vector search only | 0.713 | 83.6% | 73.0% | 18 ms |
| BM25 keyword search only | 0.652 | 77.4% | 69.3% | 10 ms |
| Vector search, then reranker (30 candidates) | 0.704 | 83.5% | 72.3% | 1.1 s |
| Vector + BM25, then reranker (30 + 30 candidates, no cut-off) | 0.697 | 82.7% | 72.3% | 1.6 s |
-
nDCG@10 is the usual score for this benchmark: 1.0 means the right abstracts are at the top of the first 10 results, 0 means none of them is there. Recall@10 is the share of right abstracts found in the first 10. Hit@3 is the share of claims with a right abstract in the first 3. Time is the median per claim. All intervals below are 95% intervals from a paired bootstrap: the claims are drawn again at random 5,000 times, and both set-ups are always scored on the same claims. They are over the 300 test claims unless stated otherwise.
-
The embedding model was changed from
all-MiniLM-L6-v2tobge-small-en-v1.5. Three small models were compared with vector search alone. The rule, fixed in advance: the best nDCG@10 on SciFact's 809 training claims wins.Embedding model Training claims Test claims FiQA (648 questions) all-MiniLM-L6-v2(the old one)0.660 0.645 0.369 e5-small-v20.708 0.687 not run bge-small-en-v1.50.747 0.713 0.403 nDCG@10 with vector search alone. On the SciFact test claims, the new model is +0.068 above the old one (interval +0.041 to +0.094) and Hit@3 goes from 68.3% to 73.0% (+4.7 points, interval +1.3 to +8.3). On FiQA (57,600 financial forum posts and everyday questions about money), which played no part in the choice, it is +0.035 (interval +0.017 to +0.053) and Hit@3 goes from 51.2% to 54.3% (+3.1 points, interval +0.2 to +6.2). The rule for FiQA, also fixed in advance, was to go back to the old model only if the new one was clearly worse there. In order of events:
bge-small-en-v1.5was first compared with the old model in a quick check whose test numbers were seen; the written rule and the third model came after that. The training-claim numbers are the same in both, so the choice does not change. -
Part of the SciFact gain comes from text length. The old model reads only the first 256 tokens (word pieces) of a text, the new one up to 512, and 71% of SciFact abstracts are longer than 256 tokens. For test claims whose answering abstract fits in 256 tokens, the gain is +0.030 (interval −0.016 to +0.076, 86 claims); for longer ones, +0.083 (+0.052 to +0.116, 214 claims). The app's passages are at most about 1,000 characters, mostly under 256 tokens, so the gain the app sees is more likely near FiQA's +0.035, where only 19% of posts are longer than 256 tokens.
-
What it costs: turning documents into vectors takes about 3 to 5 times longer (SciFact: 6.9 minutes against 2.5; FiQA: 53 against 11, on this laptop), so uploads are slower. Each passage is still stored as 384 numbers, so the index does not grow. The model is a larger download (about 135 MB against 90 MB) and needs a short prefix before each question, which the engine adds. Searching takes 18 ms per claim instead of 10 ms.
-
With the new model the reranker adds nothing here. Vector search alone scores 0.009 above vector search plus reranker (interval −0.018 to +0.035). With the old model the reranker lifted nDCG@10 by 0.046 (interval +0.016 to +0.075). BM25 adds nothing on top either (−0.007, interval −0.020 to +0.006). Both may help more on other kinds of documents; SciFact does not test that.
-
A bug found by this test. BM25 used to split text on spaces only, so "cancer." and "cancer" were different words. Splitting into words raised BM25 alone from 0.560 to 0.652 (95% interval +0.066 to +0.119); the full pipeline, then with the old embedding model, barely moved (0.694 to 0.696). The numbers before the fix are in commit
4350ee6. The new splitting was chosen while looking at SciFact, so it was also checked on a dataset that played no part in that choice: on FiQA (57,600 financial forum posts, 648 test questions), BM25 alone went from 0.159 to 0.217 nDCG@10 and Hit@3 from 23.0% to 30.9% (python -m evaluation.bm25_tokens_check --dataset fiqa, results inevaluation/bm25_tokens_check.json).
The app asks for 3 passages (9 vector + 9 BM25 candidates, put in order by the reranker) and drops passages with a reranker confidence below 0.3; when nothing is left, it says it cannot answer. It takes about 0.55 s per claim.
| Default call | Hit@3 | Claims with no passage |
|---|---|---|
| With the 0.3 cut-off (as the app runs) | 67.0% | 12.3% (37 of 300) |
| Without the cut-off | 72.3% | 0% |
- The new embedding model did not move this number. With either model the app finds a right abstract for 67.0% of the claims (difference 0, interval −1 to +1 point). Here the reranker picks the final 3 passages, so a better vector search changes the candidates but not what the app returns. Without the cut-off, the app's call (72.3%) and vector search alone (73.0%) are level (+0.7 points for vector search, interval −3.0 to +4.3).
- The cut-off costs 16 claims a right passage: 9 that get no passage at all and 7 whose list was cut to 1 or 2 passages. In all, 152 of the 300 claims get fewer than 3 passages. For the other 28 claims with no passage, the 3 passages the app would have used were all wrong, so not answering was the better outcome.
Whole abstracts are longer than the app's passages, so the share of dropped passages and the timings may differ on your own documents.
Every SciFact claim has an answer among the abstracts, so the test above shows only what the cut-off costs. To see what it saves, the claims were also run on a second index without the abstracts that SciFact marks as answering them: there, the right outcome is to give no passage.
The reranker's confidence tells the two cases apart only weakly. On the test claims, a claim with an answer in the index gets a higher top confidence than a claim without one 70% of the time (AUROC 0.70; 0.5 would be chance). The median top confidence is 0.97 with an answer and 0.71 without, so most claims without an answer still get confident passages.
| Cut-off | Hit@3 (answer in the index) | No passage given (no answer in the index) |
|---|---|---|
| none | 72.3% | 0% |
| 0.1 | 70.0% | 19.0% |
| 0.3 (the app) | 67.0% | 30.3% |
| 0.5 | 64.0% | 36.0% |
| 0.7 | 58.7% | 49.3% |
| 0.9 | 52.0% | 67.0% |
300 test claims on each index. Each step up refuses more claims without an answer, but also loses claims that had one.
Two things make the refusal shares look worse than they may be for real use. SciFact does not mark every abstract that answers a claim, so some passages counted as wrong here may in fact answer it: on the second index, 99 of the 300 claims still get a top passage with a confidence of 0.9 or more. And the passages left there are on the same subject as the claim, which makes refusing harder than for a question about something the documents do not cover at all.
The app keeps 0.3. Before running, the rule for choosing the cut-off on the training claims was set as the mean of the two columns. That rule picked 0.95, the top of the range tried, which on test refuses 44% of the claims that do have an answer (Hit@3 47.7%). The rule itself is at fault: refusing every claim scores 0.5 under it, more than answering every claim (0.38). So its choice was not adopted. Which cut-off is right depends on how often users ask about things that are not in their documents, and SciFact cannot say that.
With the new embedding model, on SciFact, it does not clearly pay for itself. Here is vector search alone (the 3 most similar abstracts, no reranker) against the app's call:
| Hit@3 | Training claims (809) | Test claims (300) |
|---|---|---|
| No cut-off: vector search alone | 79.0% | 73.0% |
| No cut-off: the app's call | 75.9% | 72.3% |
| Difference | +3.1 (+0.5 to +5.6) | +0.7 (−3.0 to +4.3) |
| With a cut-off: vector search alone | 70.1% | 64.3% |
| With a cut-off: the app's call (0.3) | 68.9% | 67.0% |
| Difference | +1.2 (−1.5 to +4.0) | −2.7 (−6.7 to +1.3) |
For vector search, the cut-off is a similarity score of 0.74 (1 means the same meaning). It was chosen on the training claims so that it refuses as many claims without an answer as the app does: on test, 30.0% against 30.3%. Its score tells the two cases apart about as well as the reranker's confidence (AUROC 0.69 against 0.70).
- Without a cut-off, vector search alone is better on the training claims and level on test. With a cut-off, the two are level on both; on test, vector search found a right abstract a little less often, but the interval includes no difference.
- Vector search alone takes 18 ms per claim, against about 550 ms for the app's call: 30 times faster.
The reranker stays the default for now. The evidence is mixed and comes from one dataset of scientific claims, while the reranker was trained on everyday web questions, which may be closer to what users ask. Whether to drop it is the next thing to test on a second dataset.
A larger reranker did not help either. Two candidates went into the app's call without the cut-off, on the 809 training claims. The rule, fixed before running: switch only if the paired interval against the current reranker is above 0.
| Reranker | Hit@3 (training claims) | Time per claim |
|---|---|---|
ms-marco-MiniLM-L-6-v2 (the app's) |
75.9% | 0.66 s |
ms-marco-MiniLM-L-12-v2 |
75.6% | 1.3 s |
bge-reranker-base |
77.9% | 3.0 s |
bge-reranker-base is 2.0 points better (interval −0.1 to +4.1 over the training claims): close,
but not clearly above 0, and 4.5 times slower. So the app's reranker stays, and the test claims
were not used for this. None of the three beats vector search alone on these claims (79.0%).
Timings come from separate runs, so the app's call shows 0.66 s here and 0.55 s above.
Reproduce (downloads the pinned benchmark and models; times on a laptop CPU):
pip install -r requirements-eval.txt
python -m evaluation.embedder_choice --dataset scifact # embedding model choice, about 16 min
python -m evaluation.embedder_choice --dataset fiqa # check on FiQA, about 65 min
python -m evaluation.retrieval_eval --dataset scifact # first table, about 25 min
python -m evaluation.compare dense dense+rerank # paired difference with 95% interval
python -m evaluation.compare dense dense --metric hit3 \
--csv-b evaluation/per_question_scifact_all-MiniLM-L6-v2.csv # new against old embedding model
python -m evaluation.cutoff_tuning # cut-off table, about 40 min
python -m evaluation.dense_cutoff # vector search with a cut-off, about 8 min
python -m evaluation.reranker_check # larger rerankers, about 75 minResults for the old embedding model: python -m evaluation.retrieval_eval --embedder all-MiniLM-L6-v2.
You need Python 3.10 or newer and Ollama.
git clone https://github.com/YsK-dev/Rag-case-study-.git
cd Rag-case-study-
# Language models
ollama serve # leave running in its own terminal
ollama pull qwen3:1.7b
ollama pull pielee/qwen3-4b-thinking-2507_q8
# Back end
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
uvicorn main:app --reload # http://localhost:8000, API page at /docsThe first start downloads the embedding model and the reranker from Hugging Face (about 230 MB in all).
Upgrading from an earlier version: the default embedding model is now bge-small-en-v1.5, and
each embedding model has its own index. The old index is kept but not used, and the app prints a
warning at start-up when the new index is empty but uploaded files exist. Upload your documents
again, or start the app with RAG_EMBEDDER=all-MiniLM-L6-v2 to keep using the old index. A
document deleted while one model is in use stays in the other model's index. Each index records
the model that built it, and the app refuses to open an index built with a different model.
Try it:
curl -F "file=@my_notes.pdf" http://localhost:8000/api/upload
curl -X POST http://localhost:8000/api/chat \
-H "Content-Type: application/json" \
-d '{"question": "What does the report say about costs?", "model": "qwen3-1.7b", "top_k": 3}'| Endpoint | Method | What it does |
|---|---|---|
/api/upload |
POST | Add a document |
/api/documents |
GET | List documents |
/api/documents/{filename} |
GET, DELETE | Download or remove one document |
/api/documents/{filename}/preview |
GET | Text of a non-PDF document, for a preview |
/api/chat |
POST | Ask a question ("stream": true for streaming) |
/api/chat/conversations |
GET | List conversations |
/api/chat/history/{conversation_id} |
GET, DELETE | Show or clear one conversation |
/api/feedback |
POST | Thumbs up or down for an answer |
/api/models |
GET | Available models |
/api/vector-stores |
GET | Available search back ends (ChromaDB, FAISS) |
/api/health |
GET | Health check |
The unit tests check the main endpoints (chat, streaming, upload, documents, history, feedback) and input checks in a light test mode that does not load any model, so they run in about a second. The tests of the search metrics run there too.
The engine tests need the full requirements. They use the real ChromaDB and FAISS with a small fake embedding model, so no model is downloaded. They check the question prefix, unit-length vectors, one index per embedding model, the refusal of an index built with another model, and the BM25 word splitting. CI runs both sets.
pip install -r requirements-dev.txt
RAG_TESTING=1 python -m pytest # light tests
pip install -r requirements.txt faiss-cpu
python -m pytest tests/test_embedders.py tests/test_bm25_tokens.py # engine teststests/eval_judge.py sends your questions to the running API and asks larger models on Groq to
score each answer from 0 to 10 for faithfulness to the sources, relevance and completeness. You
supply the question file; the API key is read from the GROQ_API_KEY environment variable.
python tests/eval_judge.py --input my_questions.json --output judge_results.jsonl| Setting | Where | Default |
|---|---|---|
| Models | MODEL_REGISTRY in main.py |
qwen3-1.7b, qwen3-4b-thinking |
| Embedding model | RAG_EMBEDDER environment variable (EMBEDDERS in src/rag_engine.py) |
bge-small-en-v1.5 |
| Reranker | src/rag_engine.py |
cross-encoder/ms-marco-MiniLM-L-6-v2 |
| Passage size | src/rag_engine.py |
about 500 characters, 100 overlap, joined up to 1,000 |
| Confidence cut-off | retrieve() in src/rag_engine.py |
0.3 |
| Passages per answer | top_k in the chat request |
3 |
To add an Ollama model, pull it and add an entry to MODEL_REGISTRY in main.py.
The docs/ folder has the design notes written during the project: the problem, the
model layer, the search layer, the API, the architecture and a comparison of the tools considered.