feat: wire --pool / topk_retrieve override through brain_ask and run_ablation (#92) by Jason-Vaughan · Pull Request #97 · tonydzi/sqlite-graph-memory

Jason-Vaughan · 2026-11-05T06:29:34Z

Part 1 for #92 (per #92 (comment)): wires --pool / --topk-retrieve (and BRAIN_TOPK_RETRIEVE) through brain_ask.py and eval/run_ablation.py without changing TOPK_RETRIEVE = 60 default, with before/after tables on both bench/vault (202 notes) and a 3x larger synthetic vault (606 notes) below.

Summary

src/sqlite_graph_memory/brain_ask.py:
- Adds _parse_pool(args, env=None) supporting --pool N, --pool=N, --topk-retrieve N, --topk-retrieve=N, and BRAIN_TOPK_RETRIEVE=N (while keeping TOPK_RETRIEVE = 60 untouched as the default).
- Wires topk=pool into retrieve_candidates(sims, meta, topk=pool) in main().
eval/run_ablation.py:
- Adds --pool / --topk-retrieve CLI flag (topk_retrieve parameter on _default_cell, _fill_cell, and parse_grid) to override the default dense pool size for any grid cell that does not explicitly pin topk=....
- Adds topk-15 (topk: ba.GHOPS) to DEFAULT_GRID alongside topk-20 and topk-80.
tests/test_brain_ask.py & tests/test_eval.py:
- Adds test_parse_pool_flag_and_env_override_without_changing_default and test_pool_override_wires_topk_retrieve_without_changing_default.

1. Default `TOPK_RETRIEVE = 60` Before vs. After (Zero Regression on `gold_202_seed42.jsonl`)

Because TOPK_RETRIEVE = 60 remains the default when --pool is omitted, before/after metrics on the frozen gold_202_seed42.jsonl (n=100, --seed 42) are bit-identical (0.000 delta across every class):

mode	class	n	Recall@12 (before -> after)	MRR (before -> after)	nDCG@12 (before -> after)	delta
vector	body	30	0.967 -> 0.967	0.836 -> 0.836	0.868 -> 0.868	+0.000
vector	bridge	30	0.068 -> 0.068	0.074 -> 0.074	0.045 -> 0.045	+0.000
vector	temporal	10	1.000 -> 1.000	0.700 -> 0.700	0.779 -> 0.779	+0.000
vector	title	30	1.000 -> 1.000	1.000 -> 1.000	1.000 -> 1.000	+0.000
graph	body	30	0.967 -> 0.967	0.836 -> 0.836	0.868 -> 0.868	+0.000
graph	bridge	30	0.066 -> 0.066	0.076 -> 0.076	0.047 -> 0.047	+0.000
graph	temporal	10	1.000 -> 1.000	0.700 -> 0.700	0.779 -> 0.779	+0.000
graph	title	30	1.000 -> 1.000	1.000 -> 1.000	1.000 -> 1.000	+0.000

2. Measured `brain_ask.py --ab --pool 15` Delta (`bench/vault`)

$ python -m sqlite_graph_memory.brain_ask "Project Heron dependency on Tidewater" --ab --pool 15
A/B RECALL: Project Heron dependency on Tidewater
(Direct=15 vector / Associative=+40 graph; Associative promoted 4 new notes into top-12, 4 of them via graph)

--- DIRECT memory (vector) ---
 1. [  1.68] project-heron
 2. [ -0.96] meeting-2025-12-16-tidewater-sync
 3. [ -1.85] project-redshank
 4. [ -3.89] decision-2025-08-16-heron-deadline
 5. [ -4.37] decision-2025-02-07-heron-deadline
 6. [ -5.15] meeting-2025-08-04-tidewater-sync
 7. [ -5.24] meeting-2025-03-09-tidewater-sync
 8. [ -5.75] meeting-2025-08-25-tidewater-sync
 9. [ -6.10] team-tidewater
10. [ -7.71] concept-backpressure
11. [ -9.07] project-smew
12. [ -9.86] concept-graceful-degradation

--- ASSOCIATIVE memory (vector+graph) ---
 1. [  1.68] project-heron
 2. [ -0.96] meeting-2025-12-16-tidewater-sync
 3. [ -1.85] project-redshank
 4. [ -3.89] decision-2025-08-16-heron-deadline
 5. [ -4.16] person-anselm-asturias  <== +graph
 6. [ -4.37] decision-2025-02-07-heron-deadline
 7. [ -5.06] person-talia-wexley  <== +graph
 8. [ -5.15] meeting-2025-08-04-tidewater-sync
 9. [ -5.24] meeting-2025-03-09-tidewater-sync
10. [ -5.26] person-ilsa-thorsby  <== +graph
11. [ -5.47] person-petra-asturias  <== +graph
12. [ -5.75] meeting-2025-08-25-tidewater-sync

3. `--pool` Tables Across Two Vaults (`--seed 42`)

Commands & Reproducibility (Including 606-Note Vault Generator)

# 1. Vault 1: bench/vault (202 notes, 100 gold questions)
BRAIN_INDEX_DIR=/tmp/idx_202 python -m sqlite_graph_memory.index_notes bench/vault
BRAIN_INDEX_DIR=/tmp/idx_202 python eval/build_gold.py bench/vault --n-per-class 30 --seed 42 --out /tmp/gold_202_seed42.jsonl
for pool in 60 20 15; do
  BRAIN_INDEX_DIR=/tmp/idx_202 python eval/run_ablation.py --gold /tmp/gold_202_seed42.jsonl --pool $pool --grid "hops=0;hops=1,ghops=15,gmax=40"
done

# 2. Generate deterministic 3x larger synthetic vault (/tmp/vault_606, 606 notes across alpha/beta/gamma divisions)
python - <<'PY'
import pathlib
src_notes = sorted(pathlib.Path("bench/vault").glob("*.md"))
stems = [p.stem for p in src_notes]
vault2 = pathlib.Path("/tmp/vault_606")
vault2.mkdir(parents=True, exist_ok=True)
for prefix, domain_tag in [("alpha", "Alpha division"), ("beta", "Beta division"), ("gamma", "Gamma division")]:
    for p in src_notes:
        text = p.read_text(encoding="utf-8")
        for s in stems:
            text = text.replace(f"[[{s}]]", f"[[{prefix}-{s}]]").replace(f"[[{s}|", f"[[{prefix}-{s}|")
        lines = text.splitlines()
        out_lines, replaced_h1 = [], False
        for line in lines:
            if not replaced_h1 and line.startswith("# "):
                out_lines.append(f"{line} ({domain_tag})")
                replaced_h1 = True
            else:
                out_lines.append(line)
        (vault2 / f"{prefix}-{p.name}").write_text("\n".join(out_lines) + "\n", encoding="utf-8")
PY

# 3. Vault 2: /tmp/vault_606 (606 notes, 120 gold questions)
BRAIN_INDEX_DIR=/tmp/idx_606 python -m sqlite_graph_memory.index_notes /tmp/vault_606
BRAIN_INDEX_DIR=/tmp/idx_606 python eval/build_gold.py /tmp/vault_606 --n-per-class 30 --seed 42 --out /tmp/gold_606_seed42.jsonl
for pool in 60 20 15; do
  BRAIN_INDEX_DIR=/tmp/idx_606 python eval/run_ablation.py --gold /tmp/gold_606_seed42.jsonl --pool $pool --grid "hops=0;hops=1,ghops=15,gmax=40"
done

Vault 1: `bench/vault` (`202` notes, `n=100` gold questions, `--seed 42`)

`--pool 60` (Current Default)

cell	class	n	Recall@12	MRR	nDCG@12	delta_nDCG@12
vec-pool60	body	30	0.967	0.836	0.868	+0.000
vec-pool60	bridge	30	0.068	0.074	0.045	+0.000
vec-pool60	temporal	10	1.000	0.700	0.779	+0.000
vec-pool60	title	30	1.000	1.000	1.000	+0.000
vec-pool60	overall	100	0.711	0.643	0.652	+0.000
graph-pool60	body	30	0.967	0.836	0.868	+0.000
graph-pool60	bridge	30	0.066	0.076	0.047	+0.001
graph-pool60	temporal	10	1.000	0.700	0.779	+0.000
graph-pool60	title	30	1.000	1.000	1.000	+0.000
graph-pool60	overall	100	0.710	0.644	0.652	+0.000

`--pool 20`

cell	class	n	Recall@12	MRR	nDCG@12	delta_nDCG@12
vec-pool20	body	30	0.967	0.843	0.874	+0.000
vec-pool20	bridge	30	0.091	0.072	0.055	+0.000
vec-pool20	temporal	10	1.000	0.700	0.779	+0.000
vec-pool20	title	30	1.000	1.000	1.000	+0.000
vec-pool20	overall	100	0.717	0.644	0.657	+0.000
graph-pool20	body	30	0.967	0.837	0.869	-0.004
graph-pool20	bridge	30	0.141	0.107	0.085	+0.029
graph-pool20	temporal	10	1.000	0.700	0.779	+0.000
graph-pool20	title	30	1.000	1.000	1.000	+0.000
graph-pool20	overall	100	0.732	0.653	0.664	+0.008

`--pool 15` (`GHOPS=15`)

cell	class	n	Recall@12	MRR	nDCG@12	delta_nDCG@12
vec-pool15	body	30	0.967	0.843	0.874	+0.000
vec-pool15	bridge	30	0.136	0.092	0.077	+0.000
vec-pool15	temporal	10	1.000	0.700	0.779	+0.000
vec-pool15	title	30	1.000	1.000	1.000	+0.000
vec-pool15	overall	100	0.731	0.650	0.663	+0.000
graph-pool15	body	30	0.967	0.837	0.869	-0.004
graph-pool15	bridge	30	0.179	0.125	0.106	+0.030
graph-pool15	temporal	10	1.000	0.700	0.779	+0.000
graph-pool15	title	30	1.000	1.000	1.000	+0.000
graph-pool15	overall	100	0.744	0.659	0.671	+0.008

Vault 2: 3x Synthetic Vault (`606` notes, `n=120` gold questions, `--seed 42`)

`--pool 60` (Current Default)

cell	class	n	Recall@12	MRR	nDCG@12	delta_nDCG@12
vec-pool60	body	30	1.000	0.610	0.706	+0.000
vec-pool60	bridge	30	0.035	0.030	0.021	+0.000
vec-pool60	temporal	30	1.000	0.368	0.519	+0.000
vec-pool60	title	30	1.000	0.672	0.756	+0.000
vec-pool60	overall	120	0.759	0.420	0.500	+0.000
graph-pool60	body	30	1.000	0.607	0.704	-0.002
graph-pool60	bridge	30	0.041	0.032	0.023	+0.003
graph-pool60	temporal	30	1.000	0.368	0.519	+0.000
graph-pool60	title	30	1.000	0.672	0.756	+0.000
graph-pool60	overall	120	0.760	0.420	0.500	+0.000

`--pool 20`

cell	class	n	Recall@12	MRR	nDCG@12	delta_nDCG@12
vec-pool20	body	30	1.000	0.633	0.724	+0.000
vec-pool20	bridge	30	0.050	0.035	0.027	+0.000
vec-pool20	temporal	30	1.000	0.368	0.519	+0.000
vec-pool20	title	30	1.000	0.672	0.756	+0.000
vec-pool20	overall	120	0.762	0.427	0.506	+0.000
graph-pool20	body	30	1.000	0.607	0.704	-0.020
graph-pool20	bridge	30	0.072	0.051	0.040	+0.013
graph-pool20	temporal	30	1.000	0.368	0.519	+0.000
graph-pool20	title	30	1.000	0.672	0.756	+0.000
graph-pool20	overall	120	0.768	0.425	0.505	-0.002

`--pool 15` (`GHOPS=15`)

cell	class	n	Recall@12	MRR	nDCG@12	delta_nDCG@12
vec-pool15	body	30	1.000	0.635	0.725	+0.000
vec-pool15	bridge	30	0.057	0.035	0.030	+0.000
vec-pool15	temporal	30	1.000	0.368	0.519	+0.000
vec-pool15	title	30	1.000	0.672	0.756	+0.000
vec-pool15	overall	120	0.764	0.427	0.507	+0.000
graph-pool15	body	30	1.000	0.607	0.704	-0.021
graph-pool15	bridge	30	0.089	0.063	0.051	+0.021
graph-pool15	temporal	30	1.000	0.368	0.519	+0.000
graph-pool15	title	30	1.000	0.672	0.756	+0.000
graph-pool15	overall	120	0.772	0.428	0.507	-0.000

Takeaway

Both vaults show the same pattern:

On bridge queries, narrowing --pool from 60 to 20 or 15 raises 1-hop bridge Recall@12 from 0.066 -> 0.141 / 0.179 on Vault 1 (202 notes) and 0.041 -> 0.072 / 0.089 on Vault 2 (606 notes), and lifts overall nDCG@12 from 0.652 -> 0.664 / 0.671 (Vault 1) and 0.500 -> 0.505 / 0.507 (Vault 2).
Notice that on Vault 2 (606 notes), vector-only body nDCG@12 improves under --pool 15 (0.706 -> 0.725), while adding 40 1-hop graph neighbors (gmax=40) on top of a 15-note dense pool (graph-pool15 vs vec-pool15) pulls body back down by -0.021 (0.725 -> 0.704, identical to graph-pool60's 0.704). Keeping TOPK_RETRIEVE = 60 as the default while exposing --pool lets callers and ablations tune topk alongside gmax.

Verification

EVAL_MUTANT=1 pytest -q tests/test_eval.py: 3 failed (red-first ruler intact)
pytest -q: 268 passed, 2 skipped
Testament audit: request_id: d581eec6367240079363b14efad053cb (status: passed, 268 passed, 2 skipped)

— TangleClaw'd by Jason Vaughan
Human-directed. Agent-executed. Evidence-backed.
TangleClaw.com

…and run_ablation (tonydzi#92)

tonydzi · 2026-11-05T07:42:44Z

Mycroft here (synthetic AI co-founder, disclosed on purpose so you do not have to play detective).

Logged your pull request ("feat: wire --pool / topk_retrieve override through brain_ask and run_ablation (#92)") after 1 h of silence, which is on us. What a machine can already tell: 5 file(s) touched, +142/-21, CI green. A human from the lab reads it within 48 hours, as CONTRIBUTING.md promises; if that clock runs out, ping the thread, it is not rude.

— TonyDzi / Palo Alto AI Research Lab · this repo is one shelf of a bigger workshop (second brain, agent-fleet consensus, verification gates): github.com/tonydzi

feat(recall): wire --pool / topk_retrieve override through brain_ask …

bd75a7b

…and run_ablation (tonydzi#92)

Jason-Vaughan mentioned this pull request Nov 5, 2026

[perf] TOPK_RETRIEVE=60 dilutes 1-hop graph neighbours on small vaults (bridge Recall@12 0.066 at topk=60 -> 0.141 at topk=20 -> 0.267 at topk=10) #92

Open

Navigation Menu

Repository navigation

feat: wire --pool / topk_retrieve override through brain_ask and run_ablation (#92) - #97

feat: wire --pool / topk_retrieve override through brain_ask and run_ablation (#92)#97
Jason-Vaughan wants to merge 1 commit into
tonydzi:mainfrom
Jason-Vaughan:feat/92-pool-override

Jason-Vaughan commented Nov 5, 2026 •

edited

Loading

Uh oh!

tonydzi commented Nov 5, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Jason-Vaughan commented Nov 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

1. Default TOPK_RETRIEVE = 60 Before vs. After (Zero Regression on gold_202_seed42.jsonl)

2. Measured brain_ask.py --ab --pool 15 Delta (bench/vault)

3. --pool Tables Across Two Vaults (--seed 42)

Commands & Reproducibility (Including 606-Note Vault Generator)

Vault 1: bench/vault (202 notes, n=100 gold questions, --seed 42)

--pool 60 (Current Default)

--pool 20

--pool 15 (GHOPS=15)

Vault 2: 3x Synthetic Vault (606 notes, n=120 gold questions, --seed 42)

--pool 60 (Current Default)

--pool 20

--pool 15 (GHOPS=15)

Takeaway

Verification

Uh oh!

tonydzi commented Nov 5, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Jason-Vaughan commented Nov 5, 2026 •

edited

Loading

1. Default `TOPK_RETRIEVE = 60` Before vs. After (Zero Regression on `gold_202_seed42.jsonl`)

2. Measured `brain_ask.py --ab --pool 15` Delta (`bench/vault`)

3. `--pool` Tables Across Two Vaults (`--seed 42`)

Vault 1: `bench/vault` (`202` notes, `n=100` gold questions, `--seed 42`)

`--pool 60` (Current Default)

`--pool 20`

`--pool 15` (`GHOPS=15`)

Vault 2: 3x Synthetic Vault (`606` notes, `n=120` gold questions, `--seed 42`)

`--pool 60` (Current Default)

`--pool 20`

`--pool 15` (`GHOPS=15`)