Skip to content

[Accuracy diff No.78、142、143] Fix accuracy diff for astype, cast and Tensor.cast API. - #73710

Closed
Glencsa wants to merge 12 commits into
PaddlePaddle:developfrom
Glencsa:fix_accuracy_diff
Closed

[Accuracy diff No.78、142、143] Fix accuracy diff for astype, cast and Tensor.cast API.#73710
Glencsa wants to merge 12 commits into
PaddlePaddle:developfrom
Glencsa:fix_accuracy_diff

Conversation

@Glencsa

@Glencsa Glencsa commented Jun 30, 2025 •
edited by lshpku
Loading

Copy link
Copy Markdown
Contributor

PR Category

Operator Mechanism

PR Types

Bug fixes

Description

Paddle CPU 相比 GPU 以及 Torch CPU/GPU 的 float32 -> float16 在接近inf时的行为不同，可通过以下代码进行测试：

import paddle, torch
data = list(range(65500, 65532))

print(paddle.to_tensor(data, dtype='float32', place='cpu').cast('float16'))
print(paddle.to_tensor(data, dtype='float32', place='gpu').cast('float16'))

print(torch.tensor(data, dtype=torch.float32, device='cpu').to(torch.float16))
print(torch.tensor(data, dtype=torch.float32, device='cuda').to(torch.float16))

输出：

Tensor(shape=[32], dtype=float16, place=Place(cpu), stop_gradient=True,
       [65472., 65472., 65472., 65472., 65504., inf.  , inf.  , inf.  , inf.  ,
        inf.  , inf.  , inf.  , inf.  , inf.  , inf.  , inf.  , inf.  , inf.  ,
        inf.  , inf.  , inf.  , inf.  , inf.  , inf.  , inf.  , inf.  , inf.  ,
        inf.  , inf.  , inf.  , inf.  , inf.  ])
Tensor(shape=[32], dtype=float16, place=Place(gpu:0), stop_gradient=True,
       [65504., 65504., 65504., 65504., 65504., 65504., 65504., 65504., 65504.,
        65504., 65504., 65504., 65504., 65504., 65504., 65504., 65504., 65504.,
        65504., 65504., inf.  , inf.  , inf.  , inf.  , inf.  , inf.  , inf.  ,
        inf.  , inf.  , inf.  , inf.  , inf.  ])
tensor([65504., 65504., 65504., 65504., 65504., 65504., 65504., 65504., 65504.,
        65504., 65504., 65504., 65504., 65504., 65504., 65504., 65504., 65504.,
        65504., 65504.,    inf,    inf,    inf,    inf,    inf,    inf,    inf,
           inf,    inf,    inf,    inf,    inf], dtype=torch.float16)
tensor([65504., 65504., 65504., 65504., 65504., 65504., 65504., 65504., 65504.,
        65504., 65504., 65504., 65504., 65504., 65504., 65504., 65504., 65504.,
        65504., 65504.,    inf,    inf,    inf,    inf,    inf,    inf,    inf,
           inf,    inf,    inf,    inf,    inf], device='cuda:0',
       dtype=torch.float16)

分析：

  • astype、cast 和 Tensor.cast 三个API调用的同一个核函数。
  • pytorch 中对于float16类型的类型，用到了FBGEMM库，是使用AVX向量化指令每16个做一次类型转换（FbgemmFloat16ConvertAvx512.cc:64），当出现上溢出或下溢出时（即 x > 65504 或 x < 65504）, paddle直接将数值定义为inf，而pytorch在边界处，65504-65520之间的数仍定义为边界值65504，大于65520之后，才将其定义为inf

因此，本PR修改代码将paddle和pytorch做对齐，增加测试数据，三个API测试结果均可通过：
image

Glencsa added 12 commits April 10, 2025 19:31
@Glencsa
add unary ops which have spmd_rule but not add in yaml file.
8dfce3a
@Glencsa
Merge branch 'spmd_test' into develop
1d129c2
@Glencsa
Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…
b9c9e6a 
… develop
@Glencsa
Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…
f24c883 
… develop
@Glencsa
Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…
746356c 
… develop
@Glencsa
Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…
efc91c5 
… develop
@Glencsa
Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…
2109cf9 
… develop
@Glencsa
Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…
773fda6 
… develop
@Glencsa
Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…
94e7bf6 
… develop
@Glencsa
Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…
8d087dd 
… develop
@Glencsa
Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…
c9088d8 
… develop
@Glencsa
fix accuracy diff.
b99309d
@paddle-bot

paddle-bot Bot commented Jun 30, 2025

Copy link
Copy Markdown

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

@paddle-bot paddle-bot Bot added the contributor External developers label Jun 30, 2025
@luotao1 luotao1 mentioned this pull request Jun 30, 2025
Closed
@luotao1 luotao1 added the HappyOpenSource Pro 进阶版快乐开源活动，更具挑战性的任务 label Jun 30, 2025
@luotao1

luotao1 commented Jun 30, 2025

Copy link
Copy Markdown
Collaborator

@Glencsa 后续该任务的PR模版都改成：

PR Category
Execute Infrastructure

PR Types
Bug fixes

@Glencsa

Glencsa commented Jun 30, 2025

Copy link
Copy Markdown
Contributor Author

@Glencsa 后续该任务的PR模版都改成：

PR Category
Execute Infrastructure

PR Types
Bug fixes

好的好的

@luotao1

luotao1 commented Jul 2, 2025

Copy link
Copy Markdown
Collaborator

不好意思，上次通知错了，本任务的PR模版为

### PR Category
Operator Mechanism

### PR Types
Bug fixes

lshpku
lshpku reviewed Jul 2, 2025
View reviewed changes
Comment thread paddle/phi/kernels/cpu/cast_impl.h
HOSTDEVICE OutT operator()(InT in) const {
if constexpr (std::is_same<OutT, phi::dtype::float16>::value) {
float val = static_cast<float>(in);
if (val > 65504.0f && val <= 65519.0f) {

@lshpku lshpku Jul 2, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

你说的16个数一起处理，和它的rounding行为有什么必然的联系吗？我看了 https://github.com/pytorch/FBGEMM/blob/3f2cd6e070b7266248ec17c77772068b90504f8f/src/FbgemmFloat16ConvertAvx512.cc#L20 代码，也没有哪里说65504-65519之间不用变成inf，我只看到它用了CPU的_mm512_cvtps_ph指令，而我们这里是GPU的，是不是设备不同行为也不同？

@Glencsa Glencsa Jul 2, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

astype, cast and Tensor.cast 这几个API也是CPU的，paddle中直接使用的static_cast，pytorch中用到了immintrin.h。这个结果的原因推测是pytorch中代码使用了immintrin.h的宏定义_MM_FROUND_TO_NEAREST_INT导致的rounding行为。函数原型：__m256i _mm512_cvtps_ph(__m512 src, const int rounding);//rounding传入的是_MM_FROUND_TO_NEAREST_INT，其他地方没找到rounding和clip相关的操作。

@lshpku lshpku Jul 3, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

改了你的PR简介，你看看有没有道理，以及测试一下修改是否影响性能

@Glencsa Glencsa Jul 3, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR简介没有问题，请问如何测试算子性能？

@lshpku lshpku Jul 7, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

我仔细想了一下，我认为这样改仍然是不对的，完整的测试应该是确保所有取值的 float32 -> float16 行为都和GPU一致，而不是只改接近INF的部分，这样改可能还有别的值有问题。

float32一共有2^32个取值，应该全部遍历一遍进行对比，如果问题根因在rounding模式上，我估计肯定还有别的值有问题，只是用整数的输入没测出来而已。

总之，这个PR还是没有说清楚本质的原因，缺乏全量的覆盖测试，我没法给通过。

@Glencsa Glencsa Jul 7, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

你说的有道理，我注意到了，我看看还有没有其他修改方法。

@paddle-ci-bot

paddle-ci-bot Bot commented Jul 8, 2025

Copy link
Copy Markdown

Sorry to inform you that b99309d's CIs have passed for more than 7 days. To prevent PR conflicts, you need to re-run all CIs manually.

@luotao1 luotao1 closed this Aug 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lshpku lshpku lshpku left review comments

Assignees

@luotao1 luotao1

@lshpku lshpku

Labels

contributor External developers HappyOpenSource Pro 进阶版快乐开源活动，更具挑战性的任务

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Glencsa @luotao1 @lshpku