Repository navigation
vision-suite: NUM_BATCH env pins the generation batch per request - #319
Merged
Merged
Conversation
The scheduler derives num_batch from num_ctx (1024 above 4096, 2048 above 32768); a gemma4 image chunk above it is decoded in pieces. NUM_BATCH lets a campaign A/B the split at one rung without moving num_ctx. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
One line in
client.py:NUM_BATCH=<n>in the environment sendsnum_batchas a Runner option, the wayIMAGE_MIN_TOKENS/IMAGE_MAX_TOKENSalready do. The scheduler otherwise derives the generation batch fromnum_ctx(1024 above 4096, 2048 above 32768), and a gemma4 image chunk above it is decoded in pieces (llama.cpp #28954's territory); this knob lets a campaign A/B the split at one rung without movingnum_ctx.Used for the measurement now recorded in the 0.34.1 task doc on #311 ("gemma4 image chunk vs. batch": no contract cost either way, 1.5–1.9× prefill in one piece, 31b's 9 px tier 4 → 3) and for the CUDA half of #313. Checked with
emit_request.py: the option lands in the payload.🤖 Generated with Claude Code