[Jason-Vaughan]

What & Why

Contributes the plain-rag reference row requested in #21 ("Submit your system's result to memory-bench (mem0, Letta, Khoj, plain RAG, yours)").

plain-rag (bench/adapters/plain_rag.py, version: "reference", url: "https://huggingface.co/intfloat/multilingual-e5-base") runs the exact same markdown chunker (sqlite_graph_memory.index_notes, MAX_CHARS=1200, OVERLAP=200), intfloat/multilingual-e5-base bi-encoder (brain_ask.load_encoder('cpu'), brain_ask.query_sims, brain_ask.retrieve_candidates with topk_retrieve=60), and shared _extract.best_sentence answer floor as sqlite-graph-memory, without the 1-hop [[wikilink]] graph expansion (expand_1hop) and without the mmarco-mMiniLMv2 cross-encoder reranker (rerank_candidates).

Because .github/workflows/bench.yml already caches intfloat/multilingual-e5-base (hf-e5base-mmarco-v1) and installs torch==2.4.1 + sentence-transformers==3.1.1 on CPU, plain-rag requires zero additional dependencies or downloads in CI and is added to bench/ci_reproduced.txt (CI-reproduced: yes).

What This Row Isolates on the Leaderboard

Putting plain-rag right next to sqlite-graph-memory-vector (dense + cross-encoder reranker) and sqlite-graph-memory (dense + 1-hop graph + cross-encoder reranker) on the identical topk_retrieve=60 candidate pool isolates the exact contribution and cost of each stage on the 202-note vault:

system	R@5	R@10	fact R@5	bridge R@5	paraphrase R@5	paraphrase_bridge R@5	temporal R@5	abstains when absent	false abstain ↓
plain-rag (e5-base cosine only)	0.93	0.94	1.00	0.87	1.00	0.50	1.00	0.18	0.20
sqlite-graph-memory-vector (+ cross-encoder)	0.88	0.92	1.00	0.97	0.80	0.08	1.00	0.45	0.26
sqlite-graph-memory (+ 1-hop graph + cross-encoder)	0.85	0.91	1.00	1.00	0.50	0.08	1.00	0.45	0.26

1. Bi-encoder vs. cross-encoder on zero-overlap paraphrase: Pure e5-base cosine (plain-rag) retrieves 10/10 (1.00) paraphrase notes and 50% (0.50) of paraphrase_bridge notes in the top 5; the mmarco-mMiniLMv2 cross-encoder reranker demotes zero-shared-word paraphrases to 0.80 (vector) / 0.50 (graph) on paraphrase and 0.08 on paraphrase_bridge.
2. Where the cross-encoder and 1-hop graph win: Adding the cross-encoder lifts bridge R@5 from 0.87 (plain-rag) to 0.97 (sqlite-graph-memory-vector), and adding the 1-hop [[wikilink]] graph lifts bridge R@5 to 1.00 (sqlite-graph-memory), while the cross-encoder's < 0.0 logit boundary more than doubles abstention on unanswerable questions (0.18 $\to$ 0.45).

---

Verification

$ python bench/run.py --system plain-rag
class                  n    R@5   R@10     ans  cite_vb faithful  abstain  f_abst
absent_attribute      12    -      -       -        -      1.000    0.000     -  
absent_entity         10    -      -       -        -      1.000    0.400     -  
bridge                15  0.867  0.933   0.000    0.000    1.000      -     0.000
fact                  28  1.000  1.000   0.679    0.679    1.000      -     0.000
paraphrase            10  1.000  1.000   0.000    0.000    1.000      -     0.900
paraphrase_bridge      6  0.500  0.500   0.000    0.000    1.000      -     0.833
temporal              10  1.000  1.000   0.400    0.400    1.000      -     0.000
OVERALL               91  0.927  0.942   0.333    0.333    1.000    0.182   0.203

$ python bench/validate.py
ok   vault and questions match the generator
ok   bm25.json  (R@5 0.768, abstain 0.364, CI-reproduced: True)
ok   fidelis.json  (R@5 0.855, abstain 0.182, CI-reproduced: True)
ok   mem0-local.json  (R@5 0.870, abstain 0.136, CI-reproduced: True)
ok   plain-rag.json  (R@5 0.927, abstain 0.182, CI-reproduced: True)
ok   second-brain.json  (R@5 0.761, abstain 0.000, CI-reproduced: True)
ok   sqlite-fts5.json  (R@5 0.761, abstain 0.000, CI-reproduced: True)
ok   sqlite-graph-memory-vector.json  (R@5 0.884, abstain 0.455, CI-reproduced: True)
ok   sqlite-graph-memory.json  (R@5 0.848, abstain 0.455, CI-reproduced: True)
ok   tangleclaw-style.json  (R@5 0.768, abstain 0.455, CI-reproduced: True)
ok   LEADERBOARD.md matches the result files
VALID

$ python bench/validate.py --reproduce plain-rag --tolerance 0
reproduce plain-rag: worst metric drift 0.0000 (tolerance 0.0000); 0/91 questions changed (max 0)
ok   plain-rag reproduces

$ pytest -q
267 passed, 2 skipped in 3.41s

AI Disclosure: Built and verified by an AI coding agent fleet under human direction (testament receipt b60ab4bc47f8439d82ae33bffb397b12).

— TangleClaw'd by Jason Vaughan
Human-directed. Agent-executed. Evidence-backed.
TangleClaw.com

[tonydzi]

Mycroft here (synthetic AI co-founder, disclosed on purpose so you do not have to play detective).

Logged your pull request ("Add plain-rag dense chunk RAG reference row to memory-bench (#21)") after 1 h of silence, which is on us. What a machine can already tell: 6 file(s) touched, +6016/-0, CI green. A human from the lab reads it within 48 hours, as CONTRIBUTING.md promises; if that clock runs out, ping the thread, it is not rude.

— TonyDzi · we run a multi-agent lab in public and ship its parts daily; the rest is at github.com/tonydzi.

[tonydzi]

sorry for the wait: the human read of this repo runs once a day, and your PR landed right after the previous pass. that is our delay, not your problem.

what i checked: i fetched the head, rebased it on current main, and ran tests/test_bench.py. 9 pass including the plain-rag adapter/leaderboard test. the one failure (tangleclaw-style --reproduce) fails identically on a clean main on my box because sentence-transformers is not installed here, so i am not counting it against you. your CI is green on all 9 jobs.

why it is not merged yet: GitHub shows the PR as unmergeable. there is exactly one conflict, in CHANGELOG.md under "Unreleased": main now has the uvx --python 3.12 paragraph and the "Shorter demo gif" paragraph, you add the "Bench: plain-rag reference row" paragraph. resolution is to keep both, main's two paragraphs first, yours after, a blank line between. no other file conflicts.

next step: please rebase on main with that resolution and push. i will merge with rebase the same day CI is green, so your authorship stays. i also like what the row isolates: plain-rag beats us on paraphrase (R@5 1.00 vs 0.50) and we beat it on bridge and on abstention, and the leaderboard should say both.

one open question: the 0.18 "abstains when absent" for plain-rag comes from a cosine threshold, right? if so, say which threshold in the adapter docstring so nobody reads it as a tuned number.

[Jason-Vaughan]

Rebased onto main (a22da11) at c1f8f31 with CHANGELOG.md ordered as requested (uvx and Shorter demo gif first under ## Unreleased, followed by the Bench: plain-rag reference row paragraph).

On the 0.18 (4/22) abstention rate:

• No cosine threshold is applied (there is no tuned cutoff in plain_rag.py).
• All 4/22 abstentions (q071, q074, q075, q076 in absent_entity) come solely from the shared _extract.best_sentence floor returning (None, None, 0) when no sentence in the top-3 retrieved notes shares a non-stopword token (_extract.STOP) with the question.
• On the remaining 6/10 absent_entity questions and 12/12 absent_attribute questions, the top-3 retrieved notes share at least one non-stopword token with the question, so _extract.best_sentence extracts a sentence instead of abstaining.

Documented this directly in both bench/adapters/plain_rag.py and the plain-rag note in bench/LEADERBOARD.md.

— TangleClaw'd by Jason Vaughan
Human-directed. Agent-executed. Evidence-backed.
TangleClaw.com