feat(recall): cap tag-score denominator to fix query-length bias - #185
Closed
jack-arturo wants to merge 6 commits into
Closed
feat(recall): cap tag-score denominator to fix query-length bias#185jack-arturo wants to merge 6 commits into
jack-arturo wants to merge 6 commits into
Conversation
Add SEARCH_RECENCY_WINDOW_DAYS (default 180) and SEARCH_RECENCY_CURVE (linear|exp, default linear) so the recency score's decay shape can be tuned via environment instead of the hardcoded 180-day linear decay. Defaults reproduce the previous behavior exactly. Also widen the Recall Quality Lab container-restart trigger from SEARCH_WEIGHT_ to SEARCH_ so sweeps of the new SEARCH_RECENCY_* vars actually restart the API container instead of silently testing the baseline. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Code-review follow-ups for the tunable recency change: - Guard SEARCH_RECENCY_WINDOW_DAYS in config.py: non-positive values now fall back to 180 via a small _positive_or_default helper (a window of 0 caused a request-time ZeroDivisionError on every recall; negative values produced unbounded scores). Unparseable values still raise, matching the neighboring float() parses. - Make scripts/browse_memories.py diagnose read SEARCH_RECENCY_WINDOW_DAYS and SEARCH_RECENCY_CURVE from env with the same validation semantics instead of hardcoding the old 180-day linear curve, and reflect the configured window/curve in the explanatory message. - De-flake test_compute_recency_score_defaults_match_legacy_behavior by monkeypatching window=180/curve=linear instead of asserting raw config values (config.py runs load_dotenv() at import, so tuned .env files leaked into the assertion). Add coverage for the non-positive window guard. - Sync stale docs: CLAUDE.md and docs/COMPARISON.md no longer claim a fixed 180-day linear decay; docs/ENVIRONMENT_VARIABLES.md weight paragraph now scopes itself to SEARCH_WEIGHT_* since the table gained non-weight rows. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adjusts recall ranking by removing query-length bias in tag scoring: instead of dividing tag token hits by the full query token count, it caps the denominator via a new SEARCH_TAG_SCORE_TOKEN_CAP config so longer queries aren’t structurally penalized.
Changes:
- Add
SEARCH_TAG_SCORE_TOKEN_CAP(default3,0= legacy behavior) to config and document it. - Update
_compute_metadata_score()tag-score calculation to use the capped denominator and clamp to<= 1.0. - Add component-level tests covering single-token behavior, capped long-query behavior, clip behavior, legacy mode, and config parsing guard.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
automem/utils/scoring.py |
Implements capped-denominator tag scoring to reduce query-length bias. |
automem/config.py |
Introduces env parsing helper and the SEARCH_TAG_SCORE_TOKEN_CAP setting with domain guard. |
docs/ENVIRONMENT_VARIABLES.md |
Documents the new tag-score token cap and its legacy sentinel behavior. |
tests/test_api_endpoints.py |
Adds targeted tests asserting new tag-score semantics and config guard behavior. |
…s apply docker compose --env-file only affects compose-file interpolation, never container environment. Since flask-api's environment block lists no SEARCH_*/RECALL_*/CONSOLIDATION_* keys, every lab harness config override was silently dropped: all historical lab A/B runs (incl. the 2026-02-17 SEARCH_WEIGHT_RELEVANCE sweep) produced bit-identical metrics (MRR=0.7995787545787546 across baseline, candidate, and all sweep values). Wire .env.bench in via env_file with required:false so the file is optional outside lab runs. environment: still wins for keys it defines, which is fine: lab configs only use SEARCH_/RECALL_/CONSOLIDATION_ prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The tag-overlap score divided token hits by the full query length, so a 1-token query earned full tag credit from a single hit while a 5-token query needed all 5 tokens to match — biasing short queries up and long queries down. Introduce SEARCH_TAG_SCORE_TOKEN_CAP (default 3): the denominator becomes min(len(query_tokens), cap), clipped to 1.0 on the way out. Setting the cap to 0 restores the legacy full-query-length denominator; negative values fall back to the default (safer than treating a typo'd negative as an intentional legacy opt-out), and unparseable values raise like the neighboring int()/float() env parses. This intentionally changes default scoring for queries longer than 3 tokens. The default-change ships behind the section-4 eval gates per the release plan; the legacy escape hatch (cap=0) is covered by tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… production-corpus regression evidence Production-corpus A/B testing (2026-06-11, 200 queries against a 10,107-memory production clone, 3-run baseline R@5 0.650-0.655 / MRR 0.429-0.433) showed every tested cap value regresses recall on ungated free-text queries: - cap=2: R@5 0.510 (-14.2pp, paired p<0.0001), MRR 0.324 - cap=3 (previous default): R@5 0.580 (-7.2pp, p=0.0002), MRR 0.361 - cap=4: R@5 0.615 (-3.7pp, p=0.0103), MRR 0.390 - cap=0 (legacy) = baseline Mechanism: on ungated free-text queries the capped denominator inflates tag scores (1 matching tag on a 12-token query: 0.083 -> 0.33), amplifying tag noise over vector/keyword evidence. A two-stack probe A/B also showed cap=3 raising top-1 scores on known-garbage negative probes by +0.04. The cap remains available as an opt-in for tag-scoped retrieval experiments; all cap-behavior tests already pin the value explicitly via monkeypatch, so no test expectations changed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
jack-arturo
force-pushed
the
feat/tag-score-token-cap
branch
from
June 11, 2026 02:40
b291ab6 to
dcea2c6
Compare
jack-arturo
force-pushed
the
feat/tunable-recency
branch
2 times, most recently
from
June 11, 2026 18:48
02f9bb9 to
8c7d737
Compare
jack-arturo
added a commit
that referenced
this pull request
Jun 11, 2026
Continuation of #185, which GitHub auto-closed (and refused to reopen) when its stacked base branch `feat/tunable-recency` was deleted on #182's merge. Identical branch and content, now rebased onto `develop` with #182 included. See #185 for the full description, validation evidence (production-corpus lab A/B that flipped the default to `SEARCH_TAG_SCORE_TOKEN_CAP=0`), and review history. Part of the develop-branch integration series for the ranking release. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
jack-arturo
added a commit
that referenced
this pull request
Jun 11, 2026
…n tag scope (#130) (#186) ## Summary Stacked on #185. **Corrected root cause for #130.** The issue hypothesized tags act as score boosters that outrank scoped results. Code inspection shows user-passed `tags` are already a hard gate on every base path (Qdrant must-filter, graph WHERE, metadata sidecar, tag-only fallback, final post-filter). What actually happened in the repro: the gate constrained the pool to flint-tagged memories, the Forge memories were *excluded*, and within the surviving pool query-independent components (importance 0.9 × 0.1 weight + recency + tag crumbs) dominated near-zero topical evidence — confident-looking garbage with no signal the pool was gated. Fixes, with tag semantics untouched: - **Relevance gate**: `evidence = max(vector, keyword, metadata, exact)`; when query tokens exist and `evidence < RECALL_RELEVANCE_GATE`, importance/confidence/recency/relevance/tag components are scaled by `evidence/gate` (linear ramp). **Ships at 0.0 = exactly current behavior**; the enabled value comes from the eval funnel (lab sweep grid 0.10–0.25 + 22-probe + negative-control gates in automem-evals). `components` now carry `evidence` + `relevance_gated`. - **`tag_scope` response diagnostics** when tags are passed: `{filtered, pool_size_hint, gated_low_evidence}` — recall is no longer silent about gating. - **Opt-in `scope_fallback=true`**: fills remaining slots with unscoped vector results flagged `outside_tag_scope: true`, appended after scoped results, with full filter parity (min_score, time, exclude_tags, edge-based current-state suppression) — only the tag scope is lifted. In-scope memories can never reappear as "outside scope" fills (guards are mutation-tested). - **Doc corrections**: MCP `tag_match` description claimed default "exact"; the API default is `prefix`. `tags` documented as a hard scope filter (use `context_tags` to boost). Per-path semantics table in `docs/API.md`. ## Testing 20 new tests including a gate-0 bit-identity matrix, linear-ramp midpoint, route-level rerank + diagnostics, fill-resurrection mutation kills, and MCP rendering of the fill flag. Full suite 523 passed, 12 skipped; Node 14/14; black + flake8 clean. Refs #130 (will update the issue with this corrected analysis). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
jack-arturo
added a commit
that referenced
this pull request
Jun 12, 2026
…nce gate, date-aware ranking (#182, #193, #186, #187, #183, #184, #188) (#194) ## Release: ranking & recall series (develop → main)⚠️ **Merge with a MERGE COMMIT — do not squash.** release-please needs the individual conventional commits below to compute the version and changelog for PR #154. ### What's in this release | PR | Change | Default behavior | |---|---|---| | #182 | `feat(recall)`: configurable recency decay window/curve | unchanged (env-gated) | | #193 (replaces #185) | `feat(recall)`: tag-score denominator cap fixes query-length bias | unchanged (`SEARCH_TAG_SCORE_TOKEN_CAP=0`) | | #186 | `fix(recall)`: relevance gate — query-independent scoring gated on topical evidence (#130) | unchanged (gate off) | | #187 | `feat(recall)`: date-aware ranking, `recency_bias=off\|on\|auto`, latest-fact selection (#158, #159) | `RECALL_RECENCY_BIAS=off`; adds deterministic timestamp tiebreak for near-ties | | #183 | `feat(benchmarks)`: failure-mode diagnosis harness + judge quota preflight | tooling only | | #184 | `fix(mcp)`: surface stored metadata + `updated_at` in detailed recall format (#111) | additive | | #188 | `feat(enrichment)`: classification fallback-rate metrics in `/enrichment/status` | additive | Plus: CI now runs on `develop` pushes/PRs; benchmark experiment log + README contribution-policy note. ### Verification evidence - **Unit/lint/npm**: 625 pytest + 16 mcp-sse-server tests green on develop head; CI green. - **Default-preserve**: recall-lab baseline on the 10k-memory production snapshot — develop defaults vs main pooled baseline identical aggregates (R@5 0.655 / R@10 0.710 / MRR 0.434 / NDCG@10 0.501). Two-stack probe run (main vs develop, defaults): 11/12 preserve-exact, remaining diffs are near-tie reorders (top-1 score deltas ≤ 5.4e-5, the #187 timestamp tiebreak). - **Full judged 500q LongMemEval** (ship config: `RECALL_RECENCY_BIAS=auto` + `temporal-answer` harness): recall@5 96.6% (483/500), accuracy 86.0% (430/500), `judge_errors=0`, `memory_ingest_failures=0`. - **Churn attribution** (targeted re-runs of all 17 churned questions on current-main-at-defaults and develop-at-defaults): 15/17 moved with #191 (already on main) — the April canonical 97.2% floor is stale; current main measures ~97.0%. Develop-at-defaults differs from current main by **1 question in 500** (a near-tie rank-5/6 flip from #187's deterministic tiebreak). Accuracy is within answerer replicate noise (identical-config reference runs flip 28/500 answers). - Full detail: `benchmarks/EXPERIMENT_LOG.md` (2026-06-11 entry) and `benchmarks/results/lme_churn17_*` + `analyze_churn17.py`. ### Opt-in features shipped OFF `RECALL_RELEVANCE_GATE` (validated at 0.40 on lab corpus; improves negative-probe precision) and `RECALL_RECENCY_BIAS=auto` (current-state query re-ranking). Neither affects default behavior; see `docs/ENVIRONMENT_VARIABLES.md`. ### After merging release-please will update PR #154 (v0.16.0); merging *that* cuts the tag and publishes the `:stable` image — the actual user-facing deploy event for Railway template users. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #182.
tag_score = token_hits / len(query_tokens)biased ranking by query length: a 1-token query got full tag credit from a single hit while a 5-token query needed all five. NewSEARCH_TAG_SCORE_TOKEN_CAP(default 3) caps the denominator:min(1.0, hits / max(min(len(tokens), cap), 1)). One-token behavior is unchanged; long queries are no longer structurally penalized.0restores the legacy full-length denominator (escape hatch); in legacy mode the clip is a mathematical no-op, so behavior is bit-identical.SEARCH_TAG_SCORE_TOKEN_CAP=0vs3on a prod snapshot) plus the automem-evals 22-probe regression gate.Testing
6 new tests (component-level via
components["tag"], incl. clip, legacy mode, and config-domain guard); full suite 503 passed, 12 skipped; black + flake8 clean. Sweepable viamake lab-sweepwith zero extra wiring.🤖 Generated with Claude Code