Tune reranker document-length cap for long-document corpora

[phernandez]

Finding

The reranker's cost is strongly document-length dependent, measured in two places with the
same default configuration (jinaai/jina-reranker-v1-tiny-en, reranker_candidates=20,
reranker_max_document_chars=0 = unbounded):

• Short notes (golden ranking corpus, test-int/semantic/test_reranker_latency.py):
  ~90 ms P50 overhead — comfortably interactive.
• Long conversation chunks (full LoCoMo run, 1,986 queries, basic-memory-benchmarks
  main-rerank-off-long-r1 vs main-rerank-on-long-r1, 2026-08-11): p95 latency rose
  from 420 ms to ~2.1 s — a ~5x p95 regression when documents are long transcript chunks
  and the full ~100-candidate pool is scored.

Quality gain at those settings is real (#951 comment: overall R@5 +5.0pp, single_hop
+9.3pp), so this is about cost, not value.

Proposed work

1. Sweep reranker_max_document_chars (e.g. 0 / 1000 / 2000 / 4000) on the LoCoMo corpus:
   measure quality (R@5/MRR per category) vs p95 latency per setting. The body-leads
   document construction (build_rerank_document) means a modest cap should keep most of
   the signal — verify that claim with data.
2. If the sweep supports it, change the default from 0 (unbounded) to the best
   quality-preserving cap, and document the tradeoff in docs/semantic-search.md.
3. Optionally probe reranker_candidates (20 vs 10) at the chosen cap for the
   latency-sensitive path.

The measurement tooling already exists: the latency benchmark in test-int/semantic/ for
short-note numbers and basic-memory-benchmarks (run-rerank-comparison.sh pattern) for
the long-document corpus.

Acceptance

• Sweep results recorded (quality vs p95 per cap setting)
• Default reranker_max_document_chars changed only if quality loss is negligible; the
  decision and numbers documented either way
• docs/semantic-search.md tuning section updated with the measured guidance

Refs: #1231 (harness + baseline numbers), #951 (quality results), #1143 (feature).

[claude]

Claude finished @phernandez's task in 40s —— View job

---

Triage

Type: Enhancement — tuning an existing config default (reranker_max_document_chars) based on measured latency/quality tradeoffs, not a defect.

Component: Search/reranker config (config_models.py, search_repository_base.py / sqlite_search_repository.py / postgres_search_repository.py). Not cloud-specific — no cloud label applied.

Priority: Medium. No correctness issue and nothing is broken — reranking is already opt-in and the quality gain (#951: R@5 +5.0pp) is real. But the ~5x p95 regression (420ms → ~2.1s) on long-document corpora is significant enough to block confidently defaulting reranking on for those workloads, so it's worth prioritizing ahead of purely cosmetic work.

Complexity: Medium. The measurement tooling already exists (test-int/semantic/test_reranker_latency.py and the basic-memory-benchmarks harness / run-rerank-comparison.sh pattern per #1231), so this is a bounded sweep-and-document task rather than new engineering: run the reranker_max_document_chars sweep (0/1000/2000/4000) on LoCoMo, record R@5/MRR vs p95 per setting, decide the default, and update docs/semantic-search.md. The optional reranker_candidates probe is a small follow-on.

Duplicates: None found. Related: #618 (original reranker feature), #1143 (landed feature), #951 (quality results referenced here), #1231 (benchmark harness/baseline this issue builds on).

Labels applied: enhancement.

[phernandez]

Sweep results (full LoCoMo, 1,986 queries per run, main @ b799b0c)

Config	R@5	MRR	single_hop R@5	mean	p95
rerank off	0.708	0.622	0.444	256ms	420ms
cap=0 (unbounded, current default)	0.758	0.719	0.537	962ms	2121ms
cap=4000	0.758	0.719	0.537	837ms	1613ms
cap=2000	0.758	0.719	0.537	958ms	2036ms
cap=1000	0.744	0.708	0.524	652ms	1215ms

Run artifacts: main-rerank-cap{1000,2000,4000}-long-r1 in basic-memory-benchmarks;
env provenance in benchmarks/cap-sweep.log.

Reading the data honestly

• Quality: caps ≥ 2000 are identical to unbounded, to three decimals, overall and
  per-category. cap=1000 costs −1.4pp R@5 / −1.1pp MRR — too aggressive.
• Latency: the differences among cap=0/2000/4000 (1613–2121ms p95, non-monotonic) are
  run-to-run noise — the sweep ran on a machine also executing other workloads. Only
  cap=1000's halving is clearly real, and it isn't worth the quality cost.
• Implication: LoCoMo's chunk lengths mostly sit under ~2000 chars, so the rerank cost
  here is candidate-count-driven (~100 candidates/query), not document-length-driven. The
  cap does not fix this corpus's latency; it bounds the pathological case — corpora with
  multi-KB documents, which is where the original 5x p95 finding (Reranker test plan: Postgres parity, live-provider smoke, quality regression harness (post-#1143) #1231) came from.

Decision

Change the default reranker_max_document_chars from 0 (unbounded) to 2000: zero
measured quality loss on a realistic corpus, deterministic worst-case bound for long-doc
corpora, and build_rerank_document leads with the matched chunk so the retained prefix
carries the signal. Docs tuning guidance updated accordingly: to reduce rerank latency,
lower reranker_candidates; the char cap only matters for very long documents.