Skip to content

[KVCache] Support Static C8 - #4568

Merged
zhoutianzi666 merged 1 commit into
PaddlePaddle:developfrom
Sunny-bot1:static_c8
Oct 23, 2025
Merged

zhoutianzi666 merged 1 commit into
PaddlePaddle:developfrom
Sunny-bot1:static_c8

Conversation

@Sunny-bot1

@Sunny-bot1 Sunny-bot1 commented Oct 23, 2025

Copy link
Copy Markdown
Collaborator

Motivation

append_decode_cache_int8_rope_qk_norm_kernel only supports dynamic C8(fp8)

Modifications

  1. add a template argument IsDynamic for append_decode_cache_int8_rope_qk_norm_kernel to decide how to obtain kv cache scale
  2. code style changed by pre-commit

Usage or Command

add quantization_config like:

    "quantization_config":{
        "dense_quant_type":"block_wise_fp8",
        "moe_quant_type":"block_wise_fp8",
        "kv_cache_quant_type":"float8_e4m3fn",
        "quantization":"mix_quant"
    }

Accuracy Tests

Checklist

  • Add at least a tag in the PR title.
    • Tag list: [[FDConfig],[APIServer],[Engine], [Scheduler], [PD Disaggregation], [Executor], [Graph Optimization], [Speculative Decoding], [RL], [Models], [Quantization], [Loader], [OP], [KVCache], [DataProcessor], [BugFix], [Docs], [CI], [Optimization], [Feature], [Benchmark], [Others], [XPU], [HPU], [GCU], [DCU], [Iluvatar], [Metax]]
    • You can add new tags based on the PR content, but the semantics must be clear.
  • Format your code, run pre-commit before commit.
  • Add unit tests. Please write the reason in this PR if no unit tests.
  • Provide accuracy results.
  • If the current PR is submitting to the release branch, make sure the PR has been submitted to the develop branch, then cherry-pick it to the release branch with the [Cherry-Pick] PR tag.

@paddle-bot

paddle-bot Bot commented Oct 23, 2025

Copy link
Copy Markdown

Thanks for your contribution!

@zhoutianzi666
zhoutianzi666 merged commit 8718fa3 into PaddlePaddle:develop Oct 23, 2025
25 of 27 checks passed
xiaoguoguo626807 pushed a commit to xiaoguoguo626807/FastDeploy that referenced this pull request May 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants