Add offline sample retrieval check for RAGAS evaluation

[FU-max-boop]

Summary

This PR adds a small offline retrieval sanity check for the bundled LightRAG RAGAS evaluation samples.

Changes:

• add sample_retrieval_oracle.json to map each sample question to its expected sample document(s)
• add offline_retrieval_check.py, a deterministic lexical check that does not start LightRAG or call API/model/RAGAS services
• add regression coverage for the checker and bundled sample oracle
• document the optional check in the RAGAS evaluation README

Why

The existing evaluation flow can require a running LightRAG server, indexed documents, embeddings, LLM calls, and RAGAS. Before spending that setup time or API budget, it is useful to verify that the bundled sample questions are structurally aligned with the bundled sample documents.

This check catches a narrower failure mode: sample questions whose expected documents are not retrievable even under a simple deterministic baseline. It also documents that the final bundled sample question expects two documents, so top-1 retrieval only gives partial recall while top-2 gives full recall.

Validation

From the repo root:

python3 -m py_compile lightrag/evaluation/offline_retrieval_check.py tests/test_evaluation_offline_retrieval_check.py
python3 lightrag/evaluation/offline_retrieval_check.py --strict
python3 -m unittest tests/test_evaluation_offline_retrieval_check.py
git diff --check

Observed result for the default strict check:

Queries: 6
Top-k: 2
Average recall@k: 1.000
Mean reciprocal rank: 1.000
Full-recall queries: 6/6
No-hit queries: 0

I also checked --top-k 1 --strict; it exits non-zero because the multi-document sample has recall@1 = 0.500, which is the intended strict behavior.

pytest and ruff are not installed in my local environment, so I could not run those commands here.

Scope / limits

This PR does not change LightRAG retrieval behavior, the API server, embeddings, LLM calls, or RAGAS scoring. It only adds an offline sample-data sanity check and test coverage for that check.

[danielaskdd]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e1def36a8d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Do not count zero-score documents as retrieved

When a sample question has no non-stopword overlap with the documents, every document still remains in ranked with score 0 and is ordered only by filename, so --strict can report full recall for an expected document that was never lexically matched (for example, an expected alpha.md passes top-1 purely because it sorts first). This masks exactly the broken sample/oracle alignment the offline check is meant to catch; zero-score documents should be excluded from recall or otherwise treated as misses.

Useful? React with 👍 / 👎.

[FU-max-boop]

Thanks for the review. I pushed b33eeab to exclude zero-score documents from the ranked retrieval list, so an expected document cannot pass purely because it sorts first by filename. I also added a regression test for the no-overlap case.

Re-ran:

• python3 -m py_compile lightrag/evaluation/offline_retrieval_check.py tests/test_evaluation_offline_retrieval_check.py
• python3 -m unittest tests/test_evaluation_offline_retrieval_check.py
• python3 lightrag/evaluation/offline_retrieval_check.py --strict
• git diff --check

[danielaskdd]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Another round soon, please!

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".