🔍 fix: exclude deferred tools from instruction token accounting - #105
Merged
Merged
Conversation
Deferred tool definitions (used for tool_search) were counted in `toolSchemaTokens` and `getTokenBudgetBreakdown().toolCount` even though they are never bound to the model until discovered. This mismatch with `getEventDrivenToolsForBinding` inflated the reported instruction token count (e.g. 292 tools for a LibreChat MCP registry) and produced spurious context-overflow errors. Both paths now go through a new `getActiveToolDefinitions()` helper that filters out `defer_loading === true` entries unless they are already in `discoveredToolNames`, matching the filter applied at bind time. Refs: LibreChat-AI/LibreChat#12702
- Correct JSDoc on getActiveToolDefinitions: it filters for token accounting, not bind-time (code_execution-only tools still pass). - Note the staleness asymmetry on getTokenBudgetBreakdown: toolCount is live after markToolsAsDiscovered but toolSchemaTokens is a snapshot. - Add regression test pinning the snapshot semantic so future changes must intentionally re-evaluate whether discovery should recompute.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
AgentContextwhere tool definitions withdefer_loading: truewere included intoolSchemaTokensandgetTokenBudgetBreakdown().toolCounteven though they are never bound to the model until discovered viatool_search.defer_loadingfilter ingetEventDrivenToolsForBindinginflated the reported instruction token count and produced spurious context-overflow errors (e.g.292 toolsin a LibreChat MCP registry).getActiveToolDefinitions()helper that filters outdefer_loading === trueentries unless they are already present indiscoveredToolNames.Refs: LibreChat-AI/LibreChat#12702
Upgrade note for consumers
If a consumer caches
toolSchemaTokensacross runs and passes it back viaAgentInputs.toolSchemaTokens(the fast-path added in 8e0ff93),calculateInstructionTokensis skipped entirely infromConfig. A cached value computed before this fix will continue to reflect the inflated count. Consumers using that cache must invalidate it when upgrading to pick up the corrected accounting.Out of scope (follow-ups worth filing)
getActiveToolDefinitions()does not filter onallowed_callers— a code_execution-only tool passed viatoolDefinitionsstill contributes totoolSchemaTokenseven though it is never bound to the model. Pre-existing; narrower fix here keeps the change focused on the issue.toolSchemaTokensis a snapshot taken duringcalculateInstructionTokensand is not recomputed onmarkToolsAsDiscovered;toolCountis live. The JSDoc ongetTokenBudgetBreakdownnow documents this, and a test pins the snapshot semantic.Test plan
npx jest src/agents/__tests__/AgentContext.test.ts— 54/54 pass, including 5 new cases:toolSchemaTokensexcludes deferred-undiscovered entriestoolSchemaTokensincludes deferred entries pre-seeded viadiscoveredToolsgetTokenBudgetBreakdown().toolCountexcludes deferred-undiscovered entriesgetTokenBudgetBreakdown().toolCountreflects newly discovered deferred toolstoolSchemaTokenssnapshot does not auto-update aftermarkToolsAsDiscoverednpx tsc --noEmit— cleannpx eslint src/agents/AgentContext.ts src/agents/__tests__/AgentContext.test.ts— cleanANTHROPIC_API_KEY/OPENAI_API_KEY), unrelated to this change