Skip to content

Count fixed request overhead in context calibration so compression doesn't underestimate - #7

Merged
nullcache merged 1 commit into
nullcache:mainfrom
uuzzrm:fix/context-calibration
Aug 17, 2026
Merged

Count fixed request overhead in context calibration so compression doesn't underestimate#7
nullcache merged 1 commit into
nullcache:mainfrom
uuzzrm:fix/context-calibration

Conversation

@uuzzrm

@uuzzrm uuzzrm commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

Found this while tracing why compression sometimes kicks in later than it should.

The calibration in ContextManager works like: estimate the messages in chars, scale by (real prompt_tokens / estimated) from the last API call. Problem is, the API bills for the system prompt and tool schemas on every request, and those aren't part of messages — so the ratio quietly absorbs that fixed overhead. That's fine while the conversation is long, but after layer 2/3 compression shrinks the message list, the fixed overhead becomes a much bigger share of the real usage. The ratio is then too big for the remaining text, measure() underestimates by 10% or more, and the 70%/90% thresholds fire late — right around where a provider's hard limit lives.

Fix: the agent registers its system prompt + serialized tool schemas as explicit fixed overhead (ContextManager.setFixedOverhead), and observe/measure count it on both sides of the ratio. The ratio stays a pure chars-per-token rate, so shrinking the conversation doesn't change what it means.

Tests:

  • fixed overhead is counted in observe/measure
  • measure stays honest after the conversation shrinks (this is the actual bug — without the fix it reads ~12 instead of ~210 after compression)
  • without setFixedOverhead the estimator behaves exactly as before
  • an agent with empty history still measures its fixed overhead

4 new tests, full suite passes.

@nullcache
nullcache merged commit a51e985 into nullcache:main Aug 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants