[Models] Qwen3VL and Qwen3VL-Moe CUDA graph Support - #5962
Merged
CSWYF3634076 merged 3 commits intoJan 9, 2026
Merged
Conversation
CSWYF3634076
had a problem deploying
to
Metax_ci
January 8, 2026 14:53 — with 
GitHub Actions
Failure
GitHub Actions
Failure
|
Thanks for your contribution! |
CSWYF3634076
changed the title
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## develop #5962 +/- ##
==========================================
Coverage ? 67.03%
==========================================
Files ? 348
Lines ? 44687
Branches ? 6876
==========================================
Hits ? 29956
Misses ? 12515
Partials ? 2216
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
gongshaotian
approved these changes
Jan 9, 2026
gongshaotian
left a comment
gongshaotian
left a comment
Collaborator
There was a problem hiding this comment.
麻烦补充下 CUDAGraph 适配前后的性能数据
Comment on lines
+154
to
+157
| # self.deepstack_input_embeds = [ | ||
| # paddle.zeros([fd_config.scheduler_config.max_num_batched_tokens, self.context_hidden_size], dtype=dtype) | ||
| # for _ in range(self.deepstack_num_level) | ||
| # ] |
Collaborator
Author
There was a problem hiding this comment.
这个注释保留主要是想如果后续框架多模单batch受到max_num_batched_tokens限制的话，后续做个替换，因为实验下来vllm是会受到max_num_batched_tokens限制的
xiaoxiaohehe001
approved these changes
Jan 9, 2026
| # Note that the current multimodal model does not limit the number of tokens | ||
| # per batch to max_num_batched_tokens, so we use model_config.max_model_len here | ||
| buffer_seq_len = fd_config.model_config.max_model_len | ||
| # self.deepstack_input_embeds = [ |
Collaborator
Author
There was a problem hiding this comment.
这个注释保留主要是想如果后续框架多模单batch受到max_num_batched_tokens限制的话，后续做个替换，因为实验下来vllm是会受到max_num_batched_tokens限制的
Collaborator
Author
这个QA还在测试中，后续补充到PR上，我自测能看出解码速度提升 |
chang-wenbin
pushed a commit
to chang-wenbin/FastDeploy
that referenced
this pull request
Mar 2, 2026
* [Models] add Qwen3VL and Qwen3VL-Moe CUDA graph support * [Models] add Qwen3VL and Qwen3VL-Moe CUDA graph support v2 * [Models] add Qwen3VL and Qwen3VL-Moe CUDA graph support v3
xiaoguoguo626807
pushed a commit
to xiaoguoguo626807/FastDeploy
that referenced
this pull request
May 7, 2026
* [Models] add Qwen3VL and Qwen3VL-Moe CUDA graph support * [Models] add Qwen3VL and Qwen3VL-Moe CUDA graph support v2 * [Models] add Qwen3VL and Qwen3VL-Moe CUDA graph support v3
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
add Qwen3VL and Qwen3VL-Moe CUDA graph support
Modifications
本PR解决Qwen3VL和 Qwen3VL-MOE启动cuda graph的问题
1.多模长上下文请求后导致后续请求重复和乱码
原因为当前多模请求单batch token没有受到max-num-batched-tokens（默认8192）限制，叠加Qwen3VL self.deepstack_input_embeds在初始化时使用了该值作为开辟固定内存的参数。在上下文超过8192时触发了self.deepstack_input_embeds扩容重新分配，新地址未进graph，导致重复和乱码。通过使用max_model_len初始化开辟固定内存buffer
2.Qwen3VL-MOE多模长上下文先请求，多模短上下文后请求，输出看不到图片的问题
原因为VisionRotaryEmbedding中使用了self._freqs_cached，这部分内存会被重写，本应该是固定值，变成了未知的随机值，进而导致image feature精度问题，所以模型看不到图片。具体哪里重写的，并且为什么是Qwen3vlmoe特有还不清楚，不过仍然能修复问题，通过修改为直接计算freqs解决
TODO
待补充性能测试结果
Usage or Command
no
Accuracy Tests
no
Checklist
[FDConfig],[APIServer],[Engine],[Scheduler],[PD Disaggregation],[Executor],[Graph Optimization],[Speculative Decoding],[RL],[Models],[Quantization],[Loader],[OP],[KVCache],[DataProcessor],[BugFix],[Docs],[CI],[Optimization],[Feature],[Benchmark],[Others],[XPU],[HPU],[GCU],[DCU],[Iluvatar],[Metax]]pre-commitbefore commit.releasebranch, make sure the PR has been submitted to thedevelopbranch, then cherry-pick it to thereleasebranch with the[Cherry-Pick]PR tag.