Repository navigation
Give up the two guaranteed keyword rescue slots - #8
Merged
Merged
Conversation
The one retrieval change that survived a frozen-holdout read, and it is a removal rather than an addition. The slots were kept for "an exact name, date or number that the embedding missed". Measured on a 404-query corpus they cost rather than buy: dropping them is +0.008 nDCG@8 on the development half [+0.003, +0.011] and +0.008 on the frozen holdout [+0.002, +0.017]. Small, but confirmed on data it was not chosen on, which is more than any addition to this pipeline managed — recency, a kind prior, quotas, MMR and every absolute or relative floor all failed to confirm or came back worse. Two measurements explain the shape of it. Improving the arm that fills the slots, with IDF weighting plus numeric and date tokens, is +0.000 on both halves: two guaranteed places at the end cannot express a better ranking however good the arm behind them gets. And the numeric category — built precisely for the case the slots exist for — is marginally better without them, 0.803 against 0.797. The mechanism was not serving the purpose it was kept for. What is genuinely given up: a source the semantic arm never returned can no longer enter on keyword evidence alone. That is a real capability, and on this corpus it is worth less than the two slots it spends. fuse keeps the mechanism and its tests, so re-enabling it is a one-argument change if a future corpus disagrees. Unaffected is the path that matters most: when the model loses the 2.5-second race there is no semantic arm at all and fuse returns the keyword ranking in full, which testWithoutSemanticHitsTheLexicalOrderSurvives pins. Losing that race costs -0.285, two orders of magnitude more than this change is worth in either direction.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
This is the only retrieval change that survived a frozen-holdout read, and it is a removal rather than an addition.
The two tail slots were kept for "an exact name, date or number that the embedding missed". Measured on the 404-query corpus in
Tools/MemoryBench:Small, but confirmed on data it was not chosen on — which is more than any addition to this pipeline managed. Recency, a kind prior, quotas, MMR and every absolute or relative floor either failed to confirm or came back resolvably worse.
Two measurements explain the shape of it
Improving the arm that fills the slots is worth nothing. IDF weighting plus numeric and date tokens, with
fuseotherwise untouched, is +0.000 on both halves. Two guaranteed places at the end cannot express a better ranking however good the arm behind them gets. (The +0.025 that the same arm shows elsewhere in the ladder is measured inside a pipeline that has already removed the slots — and it did not confirm on the holdout either, at +0.010 [−0.004, +0.023]. It is not in this PR.)The category built for the slots is better without them.
numericreads 0.803 against 0.797. The mechanism was not serving the purpose it was kept for.What is given up
A source the semantic arm never returned can no longer enter on keyword evidence alone. That is a real capability, and on this corpus it is worth less than the two slots it spends.
SemanticRanking.fusekeeps the mechanism and all of its tests, so re-enabling it is a one-argument change if a future corpus disagrees.Unaffected is the path that matters most: when the model loses the 2.5-second race there is no semantic arm at all and
fusereturns the keyword ranking in full —testWithoutSemanticHitsTheLexicalOrderSurvivespins that. Losing that race costs −0.285, two orders of magnitude more than this change is worth in either direction.Verification
Packages/SemanticMemory: 19 tests green;fuse's rescue behaviour is untouched and still covered, including therescueSlots: 0case this now relies on.StrandmacOS andNOOPiOS), becauseapp-build.ymlis default-disabled.🤖 Generated with Claude Code