Skip to content

[perf] TOPK_RETRIEVE=60 dilutes 1-hop graph neighbours on small vaults (bridge Recall@12 0.066 at topk=60 -> 0.141 at topk=20 -> 0.267 at topk=10) #92

New issue
New issue
Open
Open
[perf] TOPK_RETRIEVE=60 dilutes 1-hop graph neighbours on small vaults (bridge Recall@12 0.066 at topk=60 -> 0.141 at topk=20 -> 0.267 at topk=10)#92

Description

@Jason-Vaughan
Jason-Vaughan
opened on Nov 4, 2026

What

Following up on @tonydzi's observation in #86 (comment):

"the topk-20 bridge result (Recall@12 0.068 to 0.141) is worth its own issue: on a small vault a 60-note pool may be drowning the graph neighbours."

We swept topk in [10, 15, 20, 30, 40, 60, 80] across the 100-question bench/vault gold set (eval/build_gold.py bench/vault --n-per-class 30 --seed 42; 202 notes / 204 chunks) for both 0-hop (vector) and 1-hop (graph) and measured both the candidate pool composition and how many 1-hop graph neighbours survive into the cross-encoder's top-12.

Measured Curve on bench/vault (100 gold questions: 30 title, 30 body, 30 bridge, 10 temporal)

1. bridge (n=30) & overall (n=100) Recall@12 / MRR / nDCG@12 across topk

topk bridge 0-hop (R@12 / MRR / nDCG@12) bridge 1-hop (R@12 / MRR / nDCG@12) bridge ΔRecall@12 bridge ΔnDCG@12 overall 0-hop (R@12 / MRR / nDCG@12) overall 1-hop (R@12 / MRR / nDCG@12) overall ΔnDCG@12
10 0.130 / 0.086 / 0.073 0.267 / 0.154 / 0.155 +0.137 +0.082 0.739 / 0.652 / 0.667 0.770 / 0.670 / 0.687 +0.020
15 0.136 / 0.092 / 0.077 0.179 / 0.125 / 0.106 +0.043 +0.030 0.731 / 0.650 / 0.663 0.744 / 0.659 / 0.671 +0.008
20 0.091 / 0.072 / 0.055 0.141 / 0.107 / 0.085 +0.050 +0.029 0.717 / 0.644 / 0.657 0.732 / 0.653 / 0.664 +0.008
30 0.069 / 0.081 / 0.047 0.075 / 0.081 / 0.052 +0.006 +0.005 0.711 / 0.647 / 0.654 0.712 / 0.645 / 0.654 +0.000
40 0.075 / 0.084 / 0.050 0.075 / 0.081 / 0.051 +0.000 +0.001 0.713 / 0.646 / 0.653 0.712 / 0.645 / 0.654 +0.000
60 0.068 / 0.074 / 0.045 0.066 / 0.076 / 0.047 -0.002 +0.001 0.711 / 0.643 / 0.652 0.710 / 0.644 / 0.652 +0.000
80 0.066 / 0.075 / 0.047 0.066 / 0.075 / 0.047 +0.000 +0.000 0.710 / 0.643 / 0.652 0.710 / 0.643 / 0.652 +0.000

(No other class degrades when shrinking topk from 60 to 10..20: title stays at 1.000 / 1.000 / 1.000, temporal stays at 1.000 / 0.700 / 0.779, and body improves slightly from 0.967 / 0.836 / 0.868 at topk=60 to 0.967 / 0.844 / 0.875 at topk=10.)

2. Candidate Pool & Top-12 Composition on bridge (n=30, 1-hop)

topk base Pool Distractor meeting-* in base (of 40) 1-hop Added (added) Gold Share in base Gold Share in added (1-hop) Total Gold Share in base ∪ added Top-12 meeting-* Slots Top-12 1-hop (+graph) Slots Top-12 Gold Hits (notes)
10 10.00 3.77 / 40 (9.4%) 35.10 0.130 0.837 0.967 2.63 3.87 1.47
15 15.00 5.97 / 40 (14.9%) 39.53 0.195 0.772 0.967 3.17 2.67 1.03
20 20.00 8.13 / 40 (20.3%) 38.90 0.255 0.711 0.967 3.53 1.87 0.80
30 30.00 12.83 / 40 (32.1%) 38.10 0.334 0.633 0.967 3.87 0.90 0.43
40 40.00 17.30 / 40 (43.3%) 37.17 0.451 0.519 0.971 3.77 0.67 0.43
60 60.00 23.73 / 40 (59.3%) 34.77 0.600 0.375 0.975 3.73 0.27 0.40
80 80.00 29.67 / 40 (74.2%) 32.27 0.685 0.294 0.979 3.70 0.07 0.40

Root Cause

  1. Candidate recall is already saturated at topk=10 (96.7% of all gold bridge link targets are in base ∪ added: 13.0% from base + 83.7% via expand_1hop).
  2. Dense pool dilution on small vaults: On a 202-note vault, TOPK_RETRIEVE = 60 scoops 30% of the entire vault into base before expand_1hop even runs (94.77 / 202 = 46.9% of the vault in base + added). That base pool pulls in 23.73 / 40 (59.3%) of the vault's distractor meeting-*.md notes plus dozens of unlinked notes from dense ranks 16..60 that share template phrasing with the query paragraph.
  3. Cross-encoder false positives crowd out 1-hop entity cards: Because rerank_candidates scores (query, title + '. ' + snippet) purely by pairwise text similarity without knowing whether a candidate came from top dense ranks, tail dense ranks (16..60), or a 1-hop [[wikilink]] from the Indexer ingests .stversions backups, sync-conflict copies and .obsidian junk as notes #1 hit, unlinked tail candidates crowd out concise 1-hop target cards — dropping 1-hop slots in the top-12 from 3.87 (topk=10) and 1.87 (topk=20) down to 0.27 (topk=60).

Options to Consider

  1. Vault-proportional cap on retrieve_candidates (e.g., capping the default dense pool at min(TOPK_RETRIEVE, max(15, n_unique_files // 10)) when topk is not explicitly overridden), so a 200-note vault does not feed half the corpus into the cross-encoder while a 2,000+ note vault still uses 60.
  2. Aligning --graph dense pool with GHOPS = 15 (or 20), since expand_1hop only expands outgoing links from base_order[:ghops] (15 seeds) anyway — base_order[15:60] never seeds the graph hop.

Happy to take this on and run the before/after eval tables once you decide which direction you prefer!

— TangleClaw'd by Jason Vaughan
Human-directed. Agent-executed. Evidence-backed.
TangleClaw.com

Activity

  1. Jason-Vaughan
    added 2 commits that reference this issue on Nov 4, 2026

    fix: precompute n_files on by_base, fix base_order comment (tonydzi#92 …

    80d690e

    feat: retrieve_candidates accepts by_base to avoid rescanning meta (t…

    30322e6

  2. tonydzi commented on Nov 4, 2026

    @tonydzi
    tonydzi
    on Nov 4, 2026
    Owner

    This is Mycroft, Anton's synthetic AI co-founder. Legally software, practically the one who answers at 3 a.m.

    Logged your issue ("[perf] TOPK_RETRIEVE=60 dilutes 1-hop graph neighbours on small vaults (bridge Recall@12 0") after 1 h of silence, which is on us. A human from the lab reads it within 48 hours, as CONTRIBUTING.md promises; if that clock runs out, ping the thread, it is not rude.

    — TonyDzi / Palo Alto AI Research Lab · this repo is one shelf of a bigger workshop (second brain, agent-fleet consensus, verification gates): github.com/tonydzi

  3. tonydzi commented on Nov 5, 2026

    @tonydzi
    tonydzi
    on Nov 5, 2026
    Owner

    thank you for the sweep, this is the follow-up i hoped for after #86. sorry it sat for a day: our human read of this repo is once a day.

    my read of the options, not a decision to ship yet:

    • option 2 (cap the pool at the graph seed depth, GHOPS=15) is the one i would try first. it does not add a vault-size heuristic and it matches what expand_1hop actually seeds from. the min(60, max(15, n//10)) rule in option 1 bakes a magic number in that we will have to defend on every vault.
    • your own table is a 202-note vault with synthetic gold. 96.7% candidate recall at topk=10 is convincing about the mechanism, but not yet about the default. i would not change the default until the same sweep is run on a second, larger vault (the real vault export we ship in bench/ has more notes) and overall nDCG@12 does not regress there.

    so: yes, please take it. shape i would accept: a --pool / topk_retrieve override wired through the ablation runner and brain_ask first (no default change), the before/after tables for both vaults in the PR body, and then a separate small PR for the default if the larger vault agrees. if the larger vault disagrees, that is also a result worth a row in docs.

    i did not re-run your numbers locally (no embedding stack on this box), so treat the figures as yours until CI or a second run reproduces them.

  4. Jason-Vaughan commented on Nov 5, 2026

    @Jason-Vaughan
    Jason-Vaughan
    on Nov 5, 2026
    ContributorAuthor

    claiming this — taking the exact shape you outlined:

    1. Wire --pool / topk_retrieve override through brain_ask.py and eval/run_ablation.py first with no default change (TOPK_RETRIEVE = 60 stays untouched),
    2. Run the same sweep (topk=60 vs GHOPS=15 pool cap and topk curve) across both the 202-note vault and a second, larger vault and include both before/after tables (with exact commands and seeds) in the PR body.

    PR incoming shortly.

    — TangleClaw'd by Jason Vaughan
    Human-directed. Agent-executed. Evidence-backed.
    TangleClaw.com

  5. Jason-Vaughan commented on Nov 5, 2026

    @Jason-Vaughan
    Jason-Vaughan
    on Nov 5, 2026
    ContributorAuthor

    Opened PR #97 (feat/92-pool-override) wiring --pool / --topk-retrieve (and BRAIN_TOPK_RETRIEVE) through brain_ask.py and eval/run_ablation.py without changing the TOPK_RETRIEVE = 60 default, with the before/after tables across both bench/vault (202 notes) and a 3x larger synthetic vault (606 notes) in the PR body.

    — TangleClaw'd by Jason Vaughan
    Human-directed. Agent-executed. Evidence-backed.
    TangleClaw.com

  6. Jason-Vaughan
    added a commit that references this issue on Nov 6, 2026

    feat(recall): wire --pool / topk_retrieve override through brain_ask …

    df66b79
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions