RAG Delivery Lab

Local-first retrieval-augmented generation workbench with a visible evidence trail.

The project answers one practical question: can a small document corpus produce a response that is inspectable, citation-backed and easy to evaluate? The default path runs without an external LLM key and returns the top retrieved excerpt in explicit extractive mode.

Stack

• API: Python 3.12 · FastAPI · Pydantic v2
• Retrieval: Qdrant · FastEmbed (BAAI/bge-small-en-v1.5)
• Generation: deterministic extractive mode by default; optional OpenAI-compatible chat endpoint for Ollama-compatible local servers and other compatible providers
• UI: React · TypeScript · Vite · nginx
• Quality: pytest · Ruff · mypy · Vitest · deterministic golden evaluations · GitHub Actions

Architecture

text/Markdown
    → normalize → heading-aware chunks → FastEmbed
    → Qdrant payload + cosine vectors

question
    → query embedding → top-k retrieval + optional source filter
    → versioned prompt → extractive or configured LLM answer
    → server-validated citations + evidence response

Document and chunk IDs are deterministic SHA-256 values. Re-indexing identical content is idempotent. The API keeps source text in Qdrant payload so every answer can return the exact evidence that was retrieved.

Run locally with Docker Compose

Requirements: Docker Desktop with Compose v2.

copy .env.example .env
# Default mode is extractive; no LLM API key is required.
docker compose up --build

Open http://localhost:3000. API docs are at http://localhost:8000/docs.

The first API startup may download the configured FastEmbed model. Qdrant data is stored in the qdrant_storage Compose volume. Stop and remove local data with:

docker compose down -v

API examples

Index a text source:

curl -X POST http://localhost:8000/api/v1/documents ^
  -H "Content-Type: application/json" ^
  -d "{\"source_name\":\"runbook.md\",\"content_type\":\"text/markdown\",\"content\":\"# Retry\\n\\nRetry once after the lease expires.\"}"

Ask for an evidence-backed answer:

curl -X POST http://localhost:8000/api/v1/query ^
  -H "Content-Type: application/json" ^
  -d "{\"question\":\"When should a worker retry?\",\"top_k\":3}"

The response includes status, answer_mode, prompt_version, citations, evidence and a trace_id. Generated output is accepted only when its citation labels match returned evidence.

Optional generated mode

Set these environment values explicitly when an OpenAI-compatible endpoint is available:

RAG_LLM_PROVIDER=openai_compatible
RAG_LLM_BASE_URL=http://host.docker.internal:11434/v1
RAG_LLM_MODEL=llama3.2:3b
RAG_LLM_API_KEY=

Provider errors are surfaced as typed error responses. The application does not silently switch providers or log prompts, questions, document text or tokens.

Tests and evaluations

Install backend dependencies and frontend dependencies:

python -m pip install -e ".[dev]"
cd frontend && npm ci
cd ..

Run the deterministic checks:

python -m pytest backend/tests -q
python scripts/evaluate.py --dataset evals/golden.jsonl
python -m ruff check backend scripts
python -m mypy backend/app
cd frontend && npm test -- --run && npm run lint && npm run typecheck && npm run build

The golden evaluator measures Recall@1, Recall@3, citation coverage/precision, refusal correctness and prompt-injection regression for the checked-in fixture. The repository makes no production users claim and no throughput claim; evaluation output is fixture-specific and must be measured by the command above.

Real Qdrant integration and Compose smoke:

RAG_INTEGRATION=1 python -m pytest backend/tests/integration -q
docker compose config --quiet
docker compose up --build -d
python scripts/smoke.py
docker compose down -v

Security boundaries

• Only JSON text/Markdown ingestion is accepted; there is no URL fetch or file-path ingestion.
• Source names, document size, chunk count, question size, top-k and context budget are bounded.
• Retrieved text is untrusted prompt data and is separated from system instructions.
• React renders answer/evidence as escaped text, not HTML.
• Qdrant is private to the Compose network; CORS is explicit.
• Secrets belong in environment variables and are absent from .env.example, logs and git history.
• The default CI mode does not need an external model credential.

Scope boundaries

This is a focused delivery lab, not an agent platform, crawler, auth system, multi-tenant service or claim of universal RAG quality. It intentionally keeps the corpus small and the retrieval path readable so changes can be evaluated before they are published.

See AGENTS.md, docs/ai-workflow.md and the committed architecture spec for the development workflow and invariants.