Repository navigation
fix(recommend): unify effective-gate units to real chars (CJK regression) - #81
Merged
Merged
Conversation
The apply-side minCompressRange gate counts raw characters (msg.text.length), but the recommend-side gates approximated chars with tokens*4 — only equivalent under the default chars/4 estimator. Hosts injecting a CJK-aware tokenizer (≈1 token/char, e.g. the kernel's own estimateTokensFast) made tokens*4 a ~4x overestimate: nudge offered ranges the kernel then atomically rejected with 'Total compressible content too small' — exactly the recommend/apply disagreement the effective gate exists to prevent. - CompressibleRange gains optional 'chars' (sum of message text lengths), always set by buildCompressibleRanges / mergeBatch - mergeRangesToThreshold batches by real chars (tail batch unchanged) - pendingByTier T1 effective filter uses chars - hand-built ranges without chars fall back to tokens*4 (backwards compat) Regression tests: CJK tokenizer below/above threshold — nudge suppression matches apply rejection; control case compresses successfully.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Review of the last 48h of merged PRs (#70/#77) found a unit mismatch between the two
minCompressRangegates:msg.text.lengthsrc/compress.tspre-checktokens * 4proxypendingByTier,mergeRangesToThresholdtokens*4 == charsonly holds for the default chars/4 estimator. The kernel explicitly encourages hosts to inject a real tokenizer (createCore({ countTokens })), and its ownestimateTokensFastis CJK-aware (≈1 token/char). With such a tokenizer:tokens*4overestimates range size ~4×compress→ apply side counts 1300 chars < 5000 → atomic rejection (Total compressible content too small)This is exactly the "recommended-then-rejected" failure mode #70's effective gate was built to eliminate — it regressed for CJK sessions. Default hosts see no change (units coincide), which is why existing tests didn't catch it.
Fix
Unify on the apply side's unit (real chars):
CompressibleRangegains optionalchars(sum of message text lengths), always set bybuildCompressibleRanges/mergeBatchmergeRangesToThresholdbatches by accumulated chars (sub-threshold tail behavior unchanged)pendingByTierT1 effective filter:(r.chars ?? r.tokens * 4) >= minCompressRangecharsfall back to the historicaltokens*4estimate — no breaking type change, legacy fixtures behave identicallyTests
New
tests/effective-gate-chars.test.ts(CJK identity tokenizer,minCompressRange: 5000):no tier has effective compressible content), andapplyCompressionof the same range rejects with "too small" — both gates now agreeapplyCompressioncreates the block — both gates agreemergeRangesToThreshold: legacy fallback (tokens: 1250w/o chars) still clears the 5000 thresholdtsc clean, build clean.