[blocked by NATIVE sglang not supporting this model yet][ATOM/ROCm] atom deepseek r1 fp4 mtp3 on mi355x

[seungrokj]

hi,

This is deepseek r1 fp4 on atom framework which supports mtp 3 tokens.

Recipe link: https://github.com/ROCm/ATOM/blob/main/recipes/DeepSeek-R1.md#mxfp4-with-mtp

Regards,
Seungrok

[github-actions]

Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook

• https://cookbook.sglang.io/
• https://docs.vllm.ai/projects/recipes/en/latest/

If it is not, please create a PR first before we can merge your PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you

• https://github.com/vllm-project/recipes
• https://github.com/sgl-project/sgl-cookbook

[github-actions]

Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook

• https://cookbook.sglang.io/
• https://docs.vllm.ai/projects/recipes/en/latest/

If it is not, please create a PR first before we can merge your PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you

• https://github.com/vllm-project/recipes
• https://github.com/sgl-project/sgl-cookbook

[seungrokj]

hi @functionstackx @cquil11
can you please review this PR?

e2e perf: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/24402497975

cc. @ChuanLi1101 @andyluo7 @chunfangamd

[cquil11]

amd/DeepSeek-R1-0528-MXFP4-MTP-MoEFP4
wait what is this?

[functionstackx]

do yall support this model in sglang...

[seungrokj]

amd/DeepSeek-R1-0528-MXFP4-MTP-MoEFP4 wait what is this?

hi @cquil11

new model 1) below, quantized shared & routed experts of layers 61.
old model 2) didn't quantized the entire layers 61.

==

1. https://huggingface.co/amd/DeepSeek-R1-0528-MXFP4-MTP-MoEFP4
   ->
   export exclude_layers="mlp.gate. *lm_head model.layers.61.eh_proj model.layers.61.shared_head.head model.layers.61.embed_tokens"
   vs
2. https://huggingface.co/amd/DeepSeek-R1-0528-MXFP4
   ->
   exclude_layers="self_attn mlp.gate. lm_head model.layers.61."

[seungrokj]

@functionstackx haven't checked mtp3 + 'native' sglang yet. (atom-oot-sglang, rocm/atom-dev:sglang-latest supports though; but same perf as atom). will check this with other folks and get back to you soon :D

[functionstackx]

@functionstackx haven't checked mtp3 + 'native' sglang yet. (atom-oot-sglang, rocm/atom-dev:sglang-latest supports though; but same perf as atom). will check this with other folks and get back to you soon :D

Yes, plz let me know when this model ckpt is supported in upstream sglang and we can accept this PR for this atom model kpt

[cquil11]

to reiterate what @functionstackx said, we think it is a good general rule of thumb to only allow this model once it is able to run on upstream SGLang image