Skip to content

Give up the two guaranteed keyword rescue slots - #8

Merged
DX23876 merged 1 commit into
mainfrom
feat/drop-guaranteed-rescue-slots
Aug 19, 2026
Merged

DX23876 merged 1 commit into
mainfrom
feat/drop-guaranteed-rescue-slots

Conversation

@DX23876

@DX23876 DX23876 commented Aug 19, 2026

Copy link
Copy Markdown
Owner

Why

This is the only retrieval change that survived a frozen-holdout read, and it is a removal rather than an addition.

The two tail slots were kept for "an exact name, date or number that the embedding missed". Measured on the 404-query corpus in Tools/MemoryBench:

Δ nDCG@8 dev Δ nDCG@8 holdout
drop the guaranteed slots +0.007 [+0.003, +0.011] +0.008 [+0.002, +0.017]

Small, but confirmed on data it was not chosen on — which is more than any addition to this pipeline managed. Recency, a kind prior, quotas, MMR and every absolute or relative floor either failed to confirm or came back resolvably worse.

Two measurements explain the shape of it

Improving the arm that fills the slots is worth nothing. IDF weighting plus numeric and date tokens, with fuse otherwise untouched, is +0.000 on both halves. Two guaranteed places at the end cannot express a better ranking however good the arm behind them gets. (The +0.025 that the same arm shows elsewhere in the ladder is measured inside a pipeline that has already removed the slots — and it did not confirm on the holdout either, at +0.010 [−0.004, +0.023]. It is not in this PR.)

The category built for the slots is better without them. numeric reads 0.803 against 0.797. The mechanism was not serving the purpose it was kept for.

What is given up

A source the semantic arm never returned can no longer enter on keyword evidence alone. That is a real capability, and on this corpus it is worth less than the two slots it spends. SemanticRanking.fuse keeps the mechanism and all of its tests, so re-enabling it is a one-argument change if a future corpus disagrees.

Unaffected is the path that matters most: when the model loses the 2.5-second race there is no semantic arm at all and fuse returns the keyword ranking in full — testWithoutSemanticHitsTheLexicalOrderSurvives pins that. Losing that race costs −0.285, two orders of magnitude more than this change is worth in either direction.

Verification

  • Packages/SemanticMemory: 19 tests green; fuse's rescue behaviour is untouched and still covered, including the rescueSlots: 0 case this now relies on.
  • Both app targets built (Strand macOS and NOOPiOS), because app-build.yml is default-disabled.
  • Full macOS suite: 2132 tests, 0 failures.
  • One line of production change; the reasoning and the numbers are in the call-site comment so the next reader does not have to find this PR.

🤖 Generated with Claude Code

The one retrieval change that survived a frozen-holdout read, and it is a removal rather than
an addition.

The slots were kept for "an exact name, date or number that the embedding missed". Measured on
a 404-query corpus they cost rather than buy: dropping them is +0.008 nDCG@8 on the development
half [+0.003, +0.011] and +0.008 on the frozen holdout [+0.002, +0.017]. Small, but confirmed
on data it was not chosen on, which is more than any addition to this pipeline managed —
recency, a kind prior, quotas, MMR and every absolute or relative floor all failed to confirm
or came back worse.

Two measurements explain the shape of it. Improving the arm that fills the slots, with IDF
weighting plus numeric and date tokens, is +0.000 on both halves: two guaranteed places at the
end cannot express a better ranking however good the arm behind them gets. And the numeric
category — built precisely for the case the slots exist for — is marginally better without them,
0.803 against 0.797. The mechanism was not serving the purpose it was kept for.

What is genuinely given up: a source the semantic arm never returned can no longer enter on
keyword evidence alone. That is a real capability, and on this corpus it is worth less than the
two slots it spends. fuse keeps the mechanism and its tests, so re-enabling it is a
one-argument change if a future corpus disagrees.

Unaffected is the path that matters most: when the model loses the 2.5-second race there is no
semantic arm at all and fuse returns the keyword ranking in full, which
testWithoutSemanticHitsTheLexicalOrderSurvives pins. Losing that race costs -0.285, two orders
of magnitude more than this change is worth in either direction.
@DX23876
DX23876 changed the base branch from feature/memory-bench to main August 19, 2026 20:48
@DX23876
DX23876 merged commit e787661 into main Aug 19, 2026
@DX23876
DX23876 deleted the feat/drop-guaranteed-rescue-slots branch August 19, 2026 20:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant