Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 16 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ on:

jobs:
unit-tests:
runs-on: ubuntu-latest
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
Expand All @@ -19,3 +19,18 @@ jobs:
- run: python -m pytest -v
env:
RAG_TESTING: "1"

engine-tests:
# The engine with its real libraries (ChromaDB, FAISS, sentence-transformers) and a small
# fake embedding model, so no model is downloaded.
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
cache: pip
cache-dependency-path: requirements.txt
- run: pip install torch --index-url https://download.pytorch.org/whl/cpu
- run: pip install -r requirements.txt faiss-cpu
- run: python -m pytest -v tests/test_embedders.py tests/test_bm25_tokens.py
212 changes: 181 additions & 31 deletions README.md

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/03_vector_layer.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ else:
|------------|---------|
| **ChromaDB** | Primary vector database |
| **FAISS** | High-performance alternative |
| **sentence-transformers** | Embedding model (all-MiniLM-L6-v2) |
| **sentence-transformers** | Embedding model (bge-small-en-v1.5; all-MiniLM-L6-v2 before) |
| **rank_bm25** | BM25 scoring for hybrid search |
| **PyMuPDF, python-docx, openpyxl** | Document parsing |

Expand Down
5 changes: 4 additions & 1 deletion docs/06_literature_review.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,13 +77,16 @@ Ollama abstracts away quantization, model management, and serving, enabling fast

| Model | Dim | Speed | Quality | Decision |
| ----------------------- | ---- | ------ | --------- | ----------- |
| **all-MiniLM-L6-v2** | 384 | Fast | Good | ✅ Default |
| all-MiniLM-L6-v2 | 384 | Fast | Good | First default |
| **bge-small-en-v1.5** | 384 | Medium | Better | ✅ Default (measured, see README) |
| all-mpnet-base-v2 | 768 | Medium | Better | Alternative |
| BGE-large | 1024 | Slow | Excellent | ❌ Memory |
| OpenAI text-embedding-3 | 1536 | API | Excellent | ❌ Cloud |

**Rationale**
MiniLM provides an excellent speed-to-quality ratio and consistently ranks well on MTEB semantic retrieval tasks while remaining CPU-friendly.
It was replaced later by bge-small-en-v1.5, which has the same vector size and searched better in this
project's own measurement (README, "How well the search works").
[see here: mteb leadorboard](https://huggingface.co/spaces/mteb/leaderboard)

---
Expand Down
7 changes: 4 additions & 3 deletions evaluation/compare.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,9 @@
python -m evaluation.compare app_default app_default_no_cutoff --metric hit3

# The same set-up in two runs, for example before and after a code change:
git show 4350ee6:evaluation/per_question_scifact.csv > before.csv
python -m evaluation.compare bm25 bm25 --csv-b before.csv
python -m evaluation.compare dense dense --csv-b evaluation/per_question_scifact_all-MiniLM-L6-v2.csv
git show 4350ee6:evaluation/per_question_scifact.csv > before.csv # before the BM25 fix
python -m evaluation.compare bm25 bm25 --csv evaluation/per_question_scifact_all-MiniLM-L6-v2.csv --csv-b before.csv
"""

import argparse
Expand All @@ -18,7 +19,7 @@
import numpy as np
import pandas as pd

DEFAULT_CSV = Path(__file__).resolve().parent / "per_question_scifact.csv"
DEFAULT_CSV = Path(__file__).resolve().parent / "per_question_scifact_bge-small-en-v1.5.csv"


def paired_bootstrap(diff: np.ndarray, resamples: int = 5000, seed: int = 0) -> tuple[float, float, float]:
Expand Down
Loading
Loading