[Intel HPU] enable tensor_wise_fp8 by fmiao2372 · Pull Request #5324 · PaddlePaddle/FastDeploy

fmiao2372 · 2025-12-02T02:53:56Z

Motivation

Enable tensor_wise_fp8 on HPU

💡 If this PR is a Cherry Pick, the PR title needs to follow the format by adding the [Cherry-Pick] label at the very beginning and appending the original PR ID at the end. For example, [Cherry-Pick][CI] Add check trigger and logic(#5191)

💡 如若此PR是Cherry Pick，PR标题需遵循格式，在最开始加上[Cherry-Pick]标签，以及最后面加上原PR ID，例如[Cherry-Pick][CI] Add check trigger and logic(#5191)

Modifications

fastdeploy/model_executor/layers/backends/intel_hpu/attention/hpu_attn_backend.py
fastdeploy/model_executor/layers/backends/intel_hpu/moe/fused_moe_hpu_backend.py
fastdeploy/model_executor/layers/backends/intel_hpu/quantization/tensor_wise_fp8.py
fastdeploy/model_executor/layers/quantization/kv_cache.py
fastdeploy/model_executor/layers/quantization/tensor_wise_fp8.py
fastdeploy/worker/hpu_model_runner.py

Usage or Command

set FD_HPU_MEASUREMENT_MODE=1, run BF16 with some datasets to do calibration and get activation abs-max values.
offline run Model_convert.py with activation abs-max values to get FP8 models. https://github.com/PaddlePaddle/PaddleCustomDevice/blob/develop/backends/intel_hpu/custom_ops/python/paddlenlp_ops/Model_convert.py
set FD_HPU_MEASUREMENT_MODE=0 and run inference.

Accuracy Tests

Checklist

[Done] Add at least a tag in the PR title.
- Tag list: [[FDConfig],[APIServer],[Engine], [Scheduler], [PD Disaggregation], [Executor], [Graph Optimization], [Speculative Decoding], [RL], [Models], [Quantization], [Loader], [OP], [KVCache], [DataProcessor], [BugFix], [Docs], [CI], [Optimization], [Feature], [Benchmark], [Others], [XPU], [HPU], [GCU], [DCU], [Iluvatar], [Metax]]
- You can add new tags based on the PR content, but the semantics must be clear.
[Done] Format your code, run pre-commit before commit.
[Done] Add unit tests. Please write the reason in this PR if no unit tests.
conducted by local tests
Provide accuracy results.
If the current PR is submitting to the release branch, make sure the PR has been submitted to the develop branch, then cherry-pick it to the release branch with the [Cherry-Pick] PR tag.

paddle-bot · 2025-12-02T02:54:08Z

Thanks for your contribution!

Copilot

Pull request overview

This PR enables tensor-wise FP8 quantization support on Intel HPU (Habana Processing Units). It introduces a measurement mode for calibration, implements HPU-specific FP8 quantization methods for linear and MoE layers, and adds support for FP8 KV cache quantization with SDPA operations.

Key Changes:

Added measurement mode controlled by FD_HPU_MEASUREMENT_MODE environment variable for collecting activation scales
Implemented HpuTensorWiseFP8LinearMethod and HpuTensorWiseFP8MoEMethod for HPU-specific FP8 quantization
Enhanced KV cache quantization with new FP8_E4M3 type and additional scale parameters for Q/S matrices

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
fastdeploy/worker/hpu_model_runner.py	Added measurement mode logic, model conversion updates, and FP8 calibration workflow
fastdeploy/platforms/intel_hpu.py	Updated attention backend import path for consistency
fastdeploy/model_executor/models/ernie4_5_moe.py	Added weight scale metadata for MoE expert layers
fastdeploy/model_executor/layers/quantization/tensor_wise_fp8.py	Added HPU-specific FP8 method routing based on platform detection
fastdeploy/model_executor/layers/quantization/kv_cache.py	Extended KV cache with FP8_E4M3 support and additional SDPA scale parameters
fastdeploy/model_executor/layers/moe/moe.py	Removed commented code line
fastdeploy/model_executor/layers/backends/intel_hpu/quantization/tensor_wise_fp8.py	New file implementing HPU tensor-wise FP8 linear quantization method
fastdeploy/model_executor/layers/backends/intel_hpu/quantization/init.py	New initialization file for HPU quantization module
fastdeploy/model_executor/layers/backends/intel_hpu/moe/fused_moe_hpu_backend.py	Added FP8 quantization support for HPU MoE with weight processing and measurement mode
fastdeploy/model_executor/layers/backends/intel_hpu/moe/init.py	Fixed docstring formatting
fastdeploy/model_executor/layers/backends/intel_hpu/attention/hpu_attn_backend.py	Integrated measurement mode and FP8 scale parameters into attention operations
fastdeploy/model_executor/layers/backends/intel_hpu/attention/init.py	Simplified module to documentation-only
fastdeploy/model_executor/layers/backends/intel_hpu/init.py	Added HpuTensorWiseFP8LinearMethod export
fastdeploy/model_executor/forward_meta.py	Added measurement_mode flag to HPUForwardMeta
fastdeploy/envs.py	Added FD_HPU_MEASUREMENT_MODE environment variable

Comments suppressed due to low confidence (1)

fastdeploy/model_executor/layers/quantization/kv_cache.py:1

The variable name max_tensor is misleading because the operation being performed is paddle.min, not paddle.max. Rename this variable to min_tensor or descale_tensor to accurately reflect the computation being performed.

"""

Copilot · 2025-12-02T02:55:05Z

            if isinstance(module, Qwen2MLP):
                module.forward = types.MethodType(fused_mlp_forward, module)


The measurement_mode attribute is set for Ernie4_5_MLP at line 332, but it is not set for Qwen2MLP before overriding its forward method. This inconsistency may cause the measurement mode logic in fused_mlp_forward to fail for Qwen2MLP. Add module.measurement_mode = measurement_mode before line 335 to ensure consistent behavior across both MLP types.

measurement mode is only supported/verified for Ernie

Copilot · 2025-12-02T02:55:05Z

+                # Take max over all logical_expert_ids (axis=0)
+                max_tensor = paddle.min(reciprocal, axis=0)
+                return max_tensor.cast(paddle.get_default_dtype())


The variable name max_tensor is misleading because the operation being performed is paddle.min, not paddle.max. Rename this variable to min_tensor or min_descale_tensor to accurately reflect the computation being performed.

Suggested change

# Take max over all logical_expert_ids (axis=0)

max_tensor = paddle.min(reciprocal, axis=0)

return max_tensor.cast(paddle.get_default_dtype())

# Take min over all logical_expert_ids (axis=0)

min_tensor = paddle.min(reciprocal, axis=0)

return min_tensor.cast(paddle.get_default_dtype())

Copilot · 2025-12-02T02:55:05Z

        self.device_id = device_id
        self.speculative_method = self.fd_config.speculative_config.method
        self.speculative_decoding = self.speculative_method is not None
+        # This measurement_mode only works in BF16 mode!


This comment indicates an important constraint, but it lacks explanation. Clarify why measurement mode only works in BF16 mode or add a reference to documentation that explains this limitation to help developers understand the restriction.

Suggested change

# This measurement_mode only works in BF16 mode!

# Measurement mode only works in BF16 mode because HPU hardware and associated libraries (e.g., SynapseAI) provide measurement and profiling features exclusively for BF16 precision.

# For more details, see: https://docs.habana.ai/en/latest/SupportedFrameworks/PaddlePaddle/PaddlePaddle.html#measurement-mode-and-bf16

Comment is not helpful

Copilot · 2025-12-02T02:55:06Z

+        act_scale = get_tensor(state_dict.pop(layer.act_scale_key))
+        act_scale = 1.0 / act_scale
+
+        # quant_weight = quant_weight.transpose([1, 0]).contiguous()


This commented-out code should either be removed if no longer needed, or uncommented with proper explanation if it's needed for future reference. Leaving unexplained commented code reduces code clarity and maintainability.

Suggested change

# quant_weight = quant_weight.transpose([1, 0]).contiguous()

codecov-commenter · 2025-12-03T02:22:11Z

Codecov Report

❌ Patch coverage is 26.31579% with 14 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (develop@c9b47f9). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
...el_executor/layers/quantization/tensor_wise_fp8.py	18.18%	7 Missing and 2 partials ⚠️
...loy/model_executor/layers/quantization/kv_cache.py	16.66%	3 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             develop    #5324   +/-   ##
==========================================
  Coverage           ?   62.11%           
==========================================
  Files              ?      329           
  Lines              ?    41566           
  Branches           ?     6349           
==========================================
  Hits               ?    25819           
  Misses             ?    13805           
  Partials           ?     1942

Flag	Coverage Δ
GPU	`62.11% <26.31%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

yuanlehome · 2025-12-04T06:14:45Z

    pre_caches_length: int = 0

+    #
+    measurement_mode: bool = False


measurement_mode是什么意思呢？

可以写在代码注释里

yuanlehome · 2025-12-04T06:16:44Z

                        **extra_weight_attrs,
                    },
                )
+            if "sdpa" in layer.cache_quant_type_str:


sdpa是什么意思的量化？除了这里，有文档对其进行解释嘛？

layer.cache_quant_type_str = "cache_fp8_sdpa_fp8" 意思是 cache 用 fp8量化，sdpa 也用fp8量化计算，也就是QK^T和 SV两个matmul是fp8的。所以除了cache 需要的k和v相关的scale，还需要q和score的相关scale.

yuanlehome · 2025-12-04T06:18:13Z


    INT8 = "int8"
    FP8 = "float8_e4m3fn"
+    FP8_E4M3 = "float8_e4m3"


是否考虑将kv_cache.py在hpu backend里新增一份单独维护？

HPU对KV_cache的改动主要集中在：
不仅仅是对cache本身做量化，在计算SDPA的过程中就做两个低精度矩阵乘。
这导致两点变化：

除了k和v需要scale，q和score同样需要scale

scale的数据类型是float32
另外一点不同是max_bound=240

如果觉得这些改动对目前代码比较大，别的厂家也没有类似需求，kv_cache今后也没有这方面扩展需要，那我们就把它单独拎出来，HPU单独维护一下

zoooo0820

这里能否参考 tests/quantization，补充下新增部分的单测；尽量保证CI覆盖率的检测

fmiao2372 · 2025-12-17T01:54:07Z

@zoooo0820 已经添加

* [Intel HPU] enable tensor_wise_fp8 * update code based on comments * fix code style issue * fix bug about RP 5138 * mv kv_cache modifications to HPU backend * fix FP8 Precision Issues * fix FP8 Precision Issues * Add quantization UT --------- Co-authored-by: yanfeich <yanfei.cheng@intel.com> Co-authored-by: YuBaoku <49938469+EmmonsCurse@users.noreply.github.com>

Copilot AI review requested due to automatic review settings December 2, 2025 02:53

Copilot AI reviewed Dec 2, 2025

View reviewed changes

yuanlehome self-assigned this Dec 2, 2025

fmiao2372 force-pushed the develop_hpu_upstream_fp8 branch from be00d48 to 4a4c5a5 Compare December 3, 2025 01:01

fmiao2372 force-pushed the develop_hpu_upstream_fp8 branch from 4a4c5a5 to 024af03 Compare December 4, 2025 02:19

yuanlehome reviewed Dec 4, 2025

View reviewed changes

fmiao2372 force-pushed the develop_hpu_upstream_fp8 branch from 024af03 to aad21bd Compare December 4, 2025 08:27

yanfeich force-pushed the develop_hpu_upstream_fp8 branch from 62960f9 to 89dea27 Compare December 5, 2025 06:19

zoooo0820 reviewed Dec 15, 2025

View reviewed changes

fmiao2372 and others added 7 commits December 16, 2025 03:29

[Intel HPU] enable tensor_wise_fp8

6d027a7

update code based on comments

b7c405a

fix code style issue

48e8153

fix bug about RP 5138

307efee

mv kv_cache modifications to HPU backend

082917d

fix FP8 Precision Issues

ae4fd32

fix FP8 Precision Issues

5aca689

fmiao2372 force-pushed the develop_hpu_upstream_fp8 branch from 02ede20 to 5aca689 Compare December 16, 2025 03:44

EmmonsCurse and others added 2 commits December 16, 2025 14:55

Merge branch 'develop' into develop_hpu_upstream_fp8

b7e8c36

Add quantization UT

cb65b09

yuanlehome approved these changes Dec 17, 2025

View reviewed changes

EmmonsCurse approved these changes Dec 17, 2025

View reviewed changes

EmmonsCurse added the skip-ci: coverage label Dec 17, 2025

zoooo0820 merged commit 404cf0e into PaddlePaddle:develop Dec 17, 2025
23 of 28 checks passed

		if isinstance(module, Qwen2MLP):
		module.forward = types.MethodType(fused_mlp_forward, module)

	# This measurement_mode only works in BF16 mode!
	# Measurement mode only works in BF16 mode because HPU hardware and associated libraries (e.g., SynapseAI) provide measurement and profiling features exclusively for BF16 precision.
	# For more details, see: https://docs.habana.ai/en/latest/SupportedFrameworks/PaddlePaddle/PaddlePaddle.html#measurement-mode-and-bf16

Conversation

fmiao2372 commented Dec 2, 2025

Motivation

Modifications

Usage or Command

Accuracy Tests

Checklist

Uh oh!

paddle-bot Bot commented Dec 2, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov-commenter commented Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zoooo0820 left a comment

Choose a reason for hiding this comment

Uh oh!

fmiao2372 commented Dec 17, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

codecov-commenter commented Dec 3, 2025 •

edited

Loading