Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
77 changes: 70 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,19 +12,78 @@ repository.
- **Reads many file types:** PDF (with tables), Word, Excel, CSV, PowerPoint, HTML, Markdown and
plain text.
- **Splits documents into passages:** text is cut into pieces of about 500 characters with 100
characters of overlap, and small neighbouring pieces are joined up to 1,000 characters. Tables,
code and formulas are kept whole. A passage that repeats, such as a page header, is kept at
most twice.
characters of overlap, and small neighbouring pieces are joined up to 1,000 characters. Tables
are kept as tables, 50 rows per passage with the header repeated. A
passage that repeats, such as a page header, is kept at most twice.
- **Finds the right passages in two steps:**
1. Candidates come from two searches: by meaning (vector search with `all-MiniLM-L6-v2` in
ChromaDB, or FAISS) and by keywords (BM25).
ChromaDB; FAISS also works after `pip install faiss-cpu`) and by keywords (BM25).
2. A cross-encoder (`ms-marco-MiniLM-L-6-v2`) reads each question–passage pair and puts the
candidates in order. Passages it is not confident about (below 0.3) are dropped. If nothing
is left, the app says it cannot answer instead of guessing.
- **Answers with a local model:** Qwen3 1.7B (fast) or Qwen3 4B Thinking, through Ollama. The
answer streams word by word and lists the passages it used.
- **Basic safety checks:** common prompt-injection phrases in questions are rejected.
- **Conversations:** follow-up questions see the last 20 messages of the conversation.
- **Conversations:** follow-up questions see the last 6 messages (3 questions and answers).

## How well the search works

Measured on [SciFact](https://huggingface.co/datasets/BeIR/scifact), a public benchmark from
BEIR: 5,183 scientific abstracts and 300 test claims (short scientific statements), each with the
abstracts that support or contradict it. Every abstract is stored whole as one passage in the
app's own search engine (`src/rag_engine.py`); reading files and splitting them into the app's
smaller passages is not part of this test. Laptop CPU (Apple M1), no GPU.

| Search set-up (10 results) | nDCG@10 | Recall@10 | Hit@3 | Time per claim |
|---|---|---|---|---|
| Vector search only | 0.645 | 78.3% | 68.3% | 10 ms |
| BM25 keyword search only | 0.652 | 77.4% | 69.3% | 10 ms |
| Vector search, then reranker (30 candidates) | 0.691 | 81.0% | 71.0% | 1.2 s |
| Vector + BM25, then reranker (30 + 30 candidates, no cut-off) | 0.696 | 83.1% | 72.3% | 1.9 s |

- **nDCG@10** is the usual score for this benchmark: 1.0 means the right abstracts are at the
top of the first 10 results, 0 means none of them is there. **Recall@10** is the share of
right abstracts found in the first 10. **Hit@3** is the share of claims with a right abstract
in the first 3. Time is the median per claim.
- **The reranker is what helps.** It lifts nDCG@10 by 0.046 over vector search alone (95%
interval +0.016 to +0.075, paired over the 300 claims), at about 1.2 s per claim on this CPU.
- **BM25 adds little on top of the reranker here:** +0.006 nDCG@10 (95% interval −0.009 to
+0.022), for about 0.6 s more per claim at 10 results. It may help more on other kinds of
documents; SciFact does not test that.
- **A bug found by this test.** BM25 used to split text on spaces only, so "cancer." and "cancer"
were different words. Splitting into words raised BM25 alone from 0.560 to 0.652 (95% interval
+0.066 to +0.119); the full pipeline barely moved (0.694 to 0.696). The numbers before the fix
are in commit `4350ee6`. The new splitting was chosen while looking at SciFact, so it was also
checked on a dataset that played no part in that choice: on
[FiQA](https://huggingface.co/datasets/BeIR/fiqa) (57,600 financial forum posts, 648 test
questions), BM25 alone went from 0.159 to 0.217 nDCG@10 and Hit@3 from 23.0% to 30.9%
(`python -m evaluation.bm25_tokens_check --dataset fiqa`, results in
`evaluation/bm25_tokens_check.json`).

**The app's default call** asks for 3 passages (9 vector + 9 BM25 candidates) and drops passages
with a reranker confidence below 0.3; when nothing is left, the app says it cannot answer. It takes
about 0.7 s per claim.

| Default call | Hit@3 | Claims with no passage |
|---|---|---|
| With the 0.3 cut-off (as the app runs) | 67.0% | 12.0% (36 of 300) |
| Without the cut-off | 72.7% | 0% |

The cut-off costs 17 claims a right passage: 10 that get no passage at all and 7 whose list was
cut to 1 or 2 passages. In all, 156 of the 300 claims get fewer than 3 passages. For the other
26 claims with no passage, the 3 passages the app would have used were all wrong, so not
answering was the better outcome. Every SciFact claim has an answer in the abstracts, so this test
shows what the cut-off costs, not how often it saves the app from answering a question the
documents cannot answer. Whole abstracts are longer than the app's passages, so the share of
dropped passages and the timings may differ on your own documents.

Reproduce (downloads the pinned benchmark and models, about 25 minutes on a laptop CPU):

```bash
pip install -r requirements-eval.txt
python -m evaluation.retrieval_eval --dataset scifact # results.json, per_question_scifact.csv
python -m evaluation.compare dense+rerank dense # paired difference with 95% interval
```

## Run it

Expand All @@ -45,6 +104,8 @@ pip install -r requirements.txt
uvicorn main:app --reload # http://localhost:8000, API page at /docs
```

The first start downloads the embedding model and the reranker from Hugging Face (about 180 MB in all).

Try it:

```bash
Expand Down Expand Up @@ -73,8 +134,10 @@ curl -X POST http://localhost:8000/api/chat \

## Tests

The unit tests check every endpoint in a light test mode that does not load any model, so they
run in about a second.
The unit tests check the main endpoints (chat, streaming, upload, documents, history, feedback)
and input checks in a light test mode that does not load any model, so they run in about a
second. The tests of the search metrics run there too; the BM25 word test runs
when the full requirements are installed.

```bash
pip install -r requirements-dev.txt
Expand Down
19 changes: 19 additions & 0 deletions evaluation/bm25_tokens_check.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
{
"fiqa": {
"dataset": "fiqa",
"documents": 57600,
"test_questions": 648,
"tokenizers": {
"split_on_spaces": {
"ndcg@10": 0.1591,
"recall@10": 0.2037,
"hit@3": 0.2299
},
"words": {
"ndcg@10": 0.2167,
"recall@10": 0.278,
"hit@3": 0.3086
}
}
}
}
64 changes: 64 additions & 0 deletions evaluation/bm25_tokens_check.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
"""BM25 alone with the old word splitting (on spaces) and the new one (words without punctuation).

The new splitting was chosen while looking at SciFact, so this script also runs on other BEIR
datasets as an independent check. It uses rank_bm25 directly, as the app does, and needs no models.

python -m evaluation.bm25_tokens_check --dataset fiqa
"""

import argparse
import json
import re
from pathlib import Path

import numpy as np
from rank_bm25 import BM25Okapi

from evaluation.metrics import hit_at_k, ndcg_at_k, recall_at_k
from evaluation.retrieval_eval import DATASETS, load

TOKENIZERS = {
"split_on_spaces": lambda text: text.lower().split(),
"words": lambda text: re.findall(r"\w+", text.lower()), # the app's bm25_tokens()
}
OUT = Path(__file__).resolve().parent / "bm25_tokens_check.json"


def main() -> None:
parser = argparse.ArgumentParser(description=__doc__.split("\n")[0])
parser.add_argument("--dataset", choices=sorted(DATASETS), default="fiqa")
args = parser.parse_args()

corpus, questions, relevant = load(args.dataset)
titles = corpus["title"].fillna("").astype(str)
bodies = corpus["text"].fillna("").astype(str)
texts = [(t + "\n" + b).strip() for t, b in zip(titles, bodies)]
ids = corpus["_id"].astype(str).tolist()
keep = [i for i, t in enumerate(texts) if t]
texts, ids = [texts[i] for i in keep], [ids[i] for i in keep]
print(f"{args.dataset}: {len(texts)} documents, {len(questions)} test questions", flush=True)

result = {"dataset": args.dataset, "documents": len(texts), "test_questions": len(questions), "tokenizers": {}}
for name, tokenize in TOKENIZERS.items():
bm25 = BM25Okapi([tokenize(t) for t in texts])
ndcg, recall, hit = [], [], []
for qid, q in questions.items():
s = bm25.get_scores(tokenize(q))
top = np.argsort(s)[::-1][:10]
ranked = [ids[i] for i in top if s[i] > 0]
ndcg.append(ndcg_at_k(ranked, relevant[qid]))
recall.append(recall_at_k(ranked, relevant[qid]))
hit.append(hit_at_k(ranked, relevant[qid], 3))
result["tokenizers"][name] = {"ndcg@10": round(float(np.mean(ndcg)), 4),
"recall@10": round(float(np.mean(recall)), 4),
"hit@3": round(float(np.mean(hit)), 4)}
print(f" {name}: {result['tokenizers'][name]}", flush=True)

saved = json.loads(OUT.read_text()) if OUT.exists() else {}
saved[args.dataset] = result
OUT.write_text(json.dumps(saved, indent=2) + "\n")
print(f"Saved {OUT}")


if __name__ == "__main__":
main()
60 changes: 60 additions & 0 deletions evaluation/compare.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
"""Compare two search set-ups question by question, with a 95% bootstrap interval.

The difference A - B is computed per question, then the questions are resampled with
replacement (5,000 times, seed 0) to get the interval.

python -m evaluation.compare dense+rerank dense
python -m evaluation.compare hybrid+rerank dense+rerank
python -m evaluation.compare app_default app_default_no_cutoff --metric hit3

# The same set-up in two runs, for example before and after a code change:
git show 4350ee6:evaluation/per_question_scifact.csv > before.csv
python -m evaluation.compare bm25 bm25 --csv-b before.csv
"""

import argparse
from pathlib import Path

import numpy as np
import pandas as pd

DEFAULT_CSV = Path(__file__).resolve().parent / "per_question_scifact.csv"


def paired_bootstrap(diff: np.ndarray, resamples: int = 5000, seed: int = 0) -> tuple[float, float, float]:
"""Mean of the per-question differences and its 95% interval."""
rng = np.random.default_rng(seed)
idx = rng.integers(0, len(diff), size=(resamples, len(diff)))
means = diff[idx].mean(axis=1)
return float(diff.mean()), float(np.percentile(means, 2.5)), float(np.percentile(means, 97.5))


def scores(csv: Path, setup: str, metric: str) -> pd.Series:
table = pd.read_csv(csv, dtype={"query_id": str})
rows = table[table["setup"] == setup]
if rows.empty:
raise SystemExit(f"No set-up '{setup}' in {csv}. Set-ups: {sorted(table['setup'].unique())}")
return rows.set_index("query_id")[metric]


def main() -> None:
parser = argparse.ArgumentParser(description=__doc__.split("\n")[0])
parser.add_argument("a")
parser.add_argument("b")
parser.add_argument("--metric", default="ndcg10", choices=["ndcg10", "recall10", "mrr10", "hit3"])
parser.add_argument("--csv", type=Path, default=DEFAULT_CSV, help="per-question results for A (and B)")
parser.add_argument("--csv-b", type=Path, help="per-question results for B, if from another run")
args = parser.parse_args()

a = scores(args.csv, args.a, args.metric)
b = scores(args.csv_b or args.csv, args.b, args.metric)
if set(a.index) != set(b.index):
raise SystemExit("The two result sets do not cover the same questions.")
diff = (a - b.reindex(a.index)).to_numpy(dtype=float)
mean, low, high = paired_bootstrap(diff)
print(f"{args.metric}: {args.a} {a.mean():.4f}, {args.b} {b.mean():.4f}, "
f"difference {mean:+.4f} (95% interval {low:+.4f} to {high:+.4f}, {len(diff)} questions)")


if __name__ == "__main__":
main()
36 changes: 36 additions & 0 deletions evaluation/metrics.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
"""Ranking metrics for one question. Plain Python, no dependencies.

`ranked` is the list of document ids the system returned, best first.
`relevant` maps each relevant document id to its relevance grade (1 or more).
"""

import math


def ndcg_at_k(ranked: list[str], relevant: dict[str, int], k: int = 10) -> float:
"""Normalised discounted cumulative gain at k, as in the BEIR benchmark."""
dcg = sum(relevant.get(doc, 0) / math.log2(rank + 2) for rank, doc in enumerate(ranked[:k]))
ideal = sorted(relevant.values(), reverse=True)[:k]
idcg = sum(grade / math.log2(rank + 2) for rank, grade in enumerate(ideal))
return dcg / idcg if idcg > 0 else 0.0


def recall_at_k(ranked: list[str], relevant: dict[str, int], k: int = 10) -> float:
"""Share of the relevant documents that appear in the top k."""
wanted = {doc for doc, grade in relevant.items() if grade > 0}
if not wanted:
return 0.0
return len(wanted & set(ranked[:k])) / len(wanted)


def hit_at_k(ranked: list[str], relevant: dict[str, int], k: int = 3) -> float:
"""1.0 if at least one relevant document is in the top k, else 0.0."""
return float(any(relevant.get(doc, 0) > 0 for doc in ranked[:k]))


def mrr_at_k(ranked: list[str], relevant: dict[str, int], k: int = 10) -> float:
"""1 / rank of the first relevant document in the top k, or 0.0 if there is none."""
for rank, doc in enumerate(ranked[:k]):
if relevant.get(doc, 0) > 0:
return 1.0 / (rank + 1)
return 0.0
Loading
Loading