Add plain-rag dense chunk RAG reference row to memory-bench (#21)

CHANGELOG.md

@@ -6,6 +6,8 @@ recorded after the fact rather than backdated to look contemporaneous.
 
 ## Unreleased
 
+**Bench: `plain-rag` reference row.** Dense cosine-only baseline (`multilingual-e5-base` over 1200-char chunks, `topk=60`, no graph hop and no cross-encoder reranker) reproduced in CI, isolating what reranking and wikilink expansion change on `memory-bench`. Thanks @Jason-Vaughan (#91), who picked up #21.
+
 ## v0.3.7 - 2026-11-05
 
 **Recall: the cross-encoder rerank is off by default.** `sgm-ask`, the MCP tool and `eval/run_eval.py` now order candidates by dense e5 similarity; `--rerank` or `SGM_RERANK=1` brings the cross-encoder back. Measured before the change: on our 184-question gold set the reranker cut body nDCG@12 from 0.815 to 0.640, and on DolphinBench data (Riley, 50 tests, 1000 sessions) Recall@5 fell from 0.61 to 0.48; it still helped short title-style queries. Dense recall is also faster on CPU (about 0.04 s against 2.8 s per query on that slice). `rerank_candidates()` takes `sims=` and keeps dense order when no cross-encoder is passed.

bench/LEADERBOARD.md

@@ -6,6 +6,7 @@ Synthetic vault: 202 notes · 91 questions (69 answerable, 22 with no answer in
 
 | system | R@5 | R@10 | answer | verbatim cite | faithful quotes | abstains when absent | false abstain ↓ | CI-reproduced | by |
 |---|---:|---:|---:|---:|---:|---:|---:|:---:|---|
+| [plain-rag](https://huggingface.co/intfloat/multilingual-e5-base) `reference` | 0.93 | 0.94 | 0.33 | 0.33 | 1.00 | 0.18 | 0.20 | yes | Jason-Vaughan |
 | [sqlite-graph-memory-vector](https://github.com/tonydzi/sqlite-graph-memory) `0.3.0` | 0.88 | 0.92 | 0.33 | 0.33 | 1.00 | 0.45 | 0.26 | yes | tonydzi |
 | [mem0-local](https://github.com/mem0ai/mem0) `2.2.1` | 0.87 | 0.92 | 0.36 | 0.36 | 1.00 | 0.14 | 0.13 | yes | tonydzi |
 | [fidelis](https://github.com/hermes-labs-ai/fidelis) `0.2.0` | 0.86 | 0.93 | 0.42 | 0.42 | 1.00 | 0.18 | 0.16 | yes | tonydzi |
@@ -19,6 +20,7 @@ Synthetic vault: 202 notes · 91 questions (69 answerable, 22 with no answer in
 
 | system | fact | bridge | paraphrase | paraphrase_bridge | temporal |
 |---|---:|---:|---:|---:|---:|
+| plain-rag | 1.00 | 0.87 | 1.00 | 0.50 | 1.00 |
 | sqlite-graph-memory-vector | 1.00 | 0.97 | 0.80 | 0.08 | 1.00 |
 | mem0-local | 1.00 | 0.70 | 1.00 | 0.25 | 1.00 |
 | fidelis | 1.00 | 0.73 | 0.90 | 0.17 | 1.00 |
@@ -32,6 +34,7 @@ Synthetic vault: 202 notes · 91 questions (69 answerable, 22 with no answer in
 
 | system | absent_entity | absent_attribute |
 |---|---:|---:|
+| plain-rag | 0.40 | 0.00 |
 | sqlite-graph-memory-vector | 0.60 | 0.33 |
 | mem0-local | 0.30 | 0.00 |
 | fidelis | 0.40 | 0.00 |
@@ -43,6 +46,7 @@ Synthetic vault: 202 notes · 91 questions (69 answerable, 22 with no answer in
 
 ## Notes from the submitters
 
+- **plain-rag** — Plain dense chunk RAG reference row: same index_notes.py chunker and intfloat/multilingual-e5-base bi-encoder as sqlite-graph-memory, ranked by cosine similarity alone (topk=60, no wikilink graph hop, no cross-encoder reranker) with the shared _extract.best_sentence floor and no cosine threshold (the 0.18 abstention rate comes solely from _extract.best_sentence returning (None, None, 0) when no sentence in the top-3 notes shares a non-stopword token with the question on 4/10 absent_entity questions). Beats sqlite-graph-memory on paraphrase (R@5 1.00 vs 0.50, where the mmarco cross-encoder demotes zero-overlap hits), while sqlite-graph-memory beats it on bridge (1.00 vs 0.87) and on abstention (0.45 vs 0.18).
 - **sqlite-graph-memory-vector** — Our own system, vector-only. Ranks ahead of graph mode on this vault (graph helps bridge, hurts paraphrase). Same answer-side weaknesses as graph mode.
 - **mem0-local** — mem0 2.x local: Ollama nomic-embed-text, embedded Qdrant, infer=False so notes are stored verbatim (extraction on would rewrite them). Pure dense search, no abstention rule. Produced and rerun in CI on Linux against a pinned Ollama container held to its haswell CPU kernel (macOS gives the same answers, R@10 0.93 there).
 - **fidelis** — fidelis-memory 0.2.0 zero_llm tier, BM25+dense+RRF with the hybrid extra on, Ollama nomic-embed-text, notes stored verbatim, embeddings memoized by the adapter (same vectors). No abstention rule at this tier. Produced and rerun in CI on Linux against a pinned Ollama container held to its haswell CPU kernel (macOS gives the same answers, a few rank-8-to-10 swaps).

bench/adapters/plain_rag.py

@@ -0,0 +1,88 @@
+# -*- coding: utf-8 -*-
+"""Plain dense chunk RAG — the no-graph, no-reranker bi-encoder reference row.
+
+Uses the exact same markdown chunker (`sqlite_graph_memory.index_notes`, MAX_CHARS=1200,
+OVERLAP=200) and `intfloat/multilingual-e5-base` bi-encoder as `sqlite-graph-memory`, plus
+the shared extractive floor in `_extract.py`, without `expand_1hop` and without cross-encoder
+reranking. Comparing this row against `sqlite-graph-memory-vector` and `sqlite-graph-memory`
+isolates what the cross-encoder reranker and the 1-hop `[[wikilink]]` graph each contribute:
+`plain-rag` beats `sqlite-graph-memory` on `paraphrase` (R@5 1.00 vs 0.50), while
+`sqlite-graph-memory` beats it on `bridge` (R@5 1.00 vs 0.87) and on abstention (0.45 vs 0.18).
+
+No cosine threshold is applied (there is no tuned cutoff): the 0.18 (`4/22`) "abstains when
+absent" rate comes solely from `_extract.best_sentence` returning `(None, None, 0)` when no
+sentence in the top-3 retrieved notes shares a non-stopword token (`_extract.STOP`) with the
+question (`q071`, `q074`, `q075`, `q076` in `absent_entity`).
+
+Runs on CPU with `CUDA_VISIBLE_DEVICES` cleared so numbers produced locally and reproduced in
+CI come from identical arithmetic.
+"""
+import os
+import subprocess
+import sys
+from pathlib import Path
+
+os.environ['CUDA_VISIBLE_DEVICES'] = ''
+ROOT = Path(__file__).resolve().parents[2]
+sys.path.insert(0, str(ROOT / 'src'))
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+from _extract import best_sentence  # noqa: E402
+
+SYSTEMS = {'plain-rag': {}}
+
+
+class Adapter:
+    """Dense chunk retrieval via e5-base cosine similarity only (no graph, no reranker)."""
+
+    def __init__(self, topk=None):
+        self._topk = topk
+
+    def info(self):
+        from sqlite_graph_memory import brain_ask as ba
+        return {
+            'version': 'reference',
+            'url': 'https://huggingface.co/intfloat/multilingual-e5-base',
+            'config': {
+                'embedder': ba.E5_MODEL,
+                'reranker': None,
+                'graph_hops': 0,
+                'topk_retrieve': self._topk or ba.TOPK_RETRIEVE,
+                'unit': 'markdown chunks via sqlite_graph_memory.index_notes (MAX_CHARS=1200)',
+                'device': 'cpu',
+                'answer': 'extractive: _extract.best_sentence of top-3 notes',
+                'abstain': 'no sentence shares a content word with the question (no reranker threshold or entity gate)',
+            },
+        }
+
+    def setup(self, vault_dir, work_dir):
+        idx = Path(work_dir) / 'index'
+        env = dict(os.environ, BRAIN_INDEX_DIR=str(idx), CUDA_VISIBLE_DEVICES='',
+                   PYTHONPATH=str(ROOT / 'src') + os.pathsep + os.environ.get('PYTHONPATH', ''))
+        subprocess.run([sys.executable, '-m', 'sqlite_graph_memory.index_notes', str(vault_dir)],
+                       check=True, env=env, stdout=subprocess.DEVNULL)
+        from sqlite_graph_memory import brain_ask as ba
+        self.ba = ba
+        self.emb, self.meta = ba.load_index(idx / '_brain_e5.npy', idx / '_brain_e5_meta.pkl')
+        self.enc = ba.load_encoder('cpu')
+        self.raw = {p.stem.lower(): p.read_text(encoding='utf-8') for p in sorted(Path(vault_dir).glob('*.md'))}
+
+    def ask(self, question):
+        ba = self.ba
+        sims = ba.query_sims(self.enc, self.emb, question)
+        cand = ba.retrieve_candidates(sims, self.meta, topk=self._topk or ba.TOPK_RETRIEVE)
+        ids, seen = [], set()
+        for i in cand:
+            nid = Path(self.meta[i]['path']).stem.lower()
+            if nid not in seen:
+                seen.add(nid)
+                ids.append(nid)
+        top = float(sims[cand[0]]) if cand else 0.0
+        nid, sent, _ = best_sentence(question, ids, self.raw.get)
+        if not sent:
+            return {'retrieved': ids, 'answer': None, 'citations': [], 'top_score': round(top, 4)}
+        return {'retrieved': ids, 'answer': sent, 'citations': [{'note': nid, 'quote': sent}],
+                'top_score': round(top, 4)}
+
+
+def make(**kwargs):
+    return Adapter(**kwargs)

bench/ci_reproduced.txt

@@ -1,6 +1,7 @@
 # Systems the bench workflow reruns from a clean checkout on every push (.github/workflows/bench.yml).
 # Only systems that need no API key and no paid service can be listed. A maintainer edits this file.
 bm25
+plain-rag          # dense chunk RAG with e5-base cosine only (no graph, no cross-encoder reranker)
 sqlite-graph-memory
 sqlite-graph-memory-vector
 sqlite-fts5