[webgpu] [ FAILED ] MatMulNBits.Float32_8b_AccuracyLevel4

[daijh]

Describe the issue

MatMulNBits.Float32_8b_AccuracyLevel4 fails on native webgpu-ep.

To reproduce

Build native webgpu-ep: commit 8aad8f9

Run:

onnxruntime_test_all.exe --gtest_filter=MatMulNBits.Float32_8b_AccuracyLevel4

Log:

Note: Google Test filter = MatMulNBits.Float32_8b_AccuracyLevel4
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from MatMulNBits
[ RUN      ] MatMulNBits.Float32_8b_AccuracyLevel4
onnxruntime\onnxruntime\test\providers\checkers.cc(392): error: The difference between cur_expected[i] and cur_actual[i] is 0.10537012666463852, which exceeds tolerance, where
cur_expected[i] evaluates to -0.22783160209655762,
cur_actual[i] evaluates to -0.1224614754319191, and
tolerance evaluates to 0.10455663502216339.
i:4662
Google Test trace:
onnxruntime\onnxruntime\test\providers\checkers.cc(568): provider type: WebGpuExecutionProvider
onnxruntime\onnxruntime\test\providers\base_tester.cc(846): registered execution providers: WebGpuExecutionProvider
onnxruntime\onnxruntime\test/common/random_generator.h(50): ORT test random seed: 1234
onnxruntime\onnxruntime\test\contrib_ops\matmul_8bits_test.cc(66): M:2, N:5120, K:3072, block_size:32, accuracy_level:4, has_zero_point:0, has_g_idx:0, has_bias:0
Stack trace:
  00007FF625A7DB08: onnxruntime::utils::MLTypeCallDispatcher<bool,float,double,unsigned char,unsigned short,unsigned int,unsigned __int64,signed char,short,int,__int64,std::basic_string<char,std::char_traits<char>,std::allocator<char> >,onnxruntime::Int4x2Base<1>,onnxruntime::Int4x2Base<0>,onnxruntime::Float8E4M3FN,onnxruntime::Float8E4M3FNUZ,onnxruntime::Float8E5M2,onnxruntime::Float8E5M2FNUZ,onnxruntime::MLFloat16,onnxruntime::BFloat16>::InvokeWithLeadingTemplateArgs<onnxruntime::test::`anonymous namespace'::TensorCheck,onnxruntime::TypeList<>,onnxruntime::Tensor const &,onnxruntime::Tensor const &,onnxruntime::test::ValidateOutputParams const &,std::basic_string<char,std::char_traits<char>,std::allocator<char> > const &>
  00007FF625A7B565: onnxruntime::test::Check<onnxruntime::Tensor>
  00007FF625A7BD5A: onnxruntime::test::CheckDispatch<onnxruntime::Tensor,std::vector<std::map<std::basic_string<char,std::char_traits<char>,std::allocator<char> >,float,std::less<std::basic_string<char,std::char_traits<char>,std::allocator<char> > >,std::allocator<std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,float> > >,std::allocator<std::map<std::basic_string<char,std::char_traits<char>,std::allocator<char> >,float,std::less<std::basic_string<char,std::char_traits<char>,std::allocator<char> > >,std::allocator<std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,float> > > > >,std::vector<std::map<__int64,float,std::less<__int64>,std::allocator<std::pair<__int64 const ,float> > >,std::allocator<std::map<__int64,float,std::less<__int64>,std::allocator<std::pair<__int64 const ,float> > > > >,onnxruntime::TensorSeq>
  00007FF625A89901: onnxruntime::test::CheckOrtValuesAreEqual
  00007FF625A714F2: onnxruntime::test::BaseTester::ExecuteModel<onnxruntime::InferenceSession>
  00007FF625A76EFC: onnxruntime::test::BaseTester::ExecuteModelForEps
  00007FF625A787F5: onnxruntime::test::BaseTester::RunWithConfig
  00007FF6259B1085: onnxruntime::test::`anonymous namespace'::RunTest8Bits<float>
  00007FF6259B4F20: onnxruntime::test::`anonymous namespace'::TestMatMul8BitsTyped<float,2,5120,3072,32,4>
  00007FF626B9E2AB: testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test,void>
  00007FF626B9E196: testing::internal::HandleExceptionsInMethodIfSupported<testing::Test,void>
  00007FF626BC4254: testing::Test::Run
  00007FF626BC436D: testing::TestInfo::Run
  00007FF626BC450E: testing::TestSuite::Run
... Google Test internal frames ...

[  FAILED  ] MatMulNBits.Float32_8b_AccuracyLevel4 (39200 ms)
[----------] 1 test from MatMulNBits (39200 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (39200 ms total)
[  PASSED  ] 0 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] MatMulNBits.Float32_8b_AccuracyLevel4

 1 FAILED TEST

Urgency

No response

Platform

Windows

OS Version

11

ONNX Runtime Installation

Built from Source

ONNX Runtime Version or Commit ID

8aad8f9

ONNX Runtime API

C++

Architecture

X64

Execution Provider

Default CPU

Execution Provider Library Version

native webgpu

[qjia7]

I ever met this issue in my machine when I enable the 8 bits matmul support. My finding is case TestMatMul8BitsTyped<float, 2, 5120, 3072, 32, 4>(); in MatMulNBits.Float32_8b_AccuracyLevel4 fails on dp4 path no matter DP4AMatMulNBitsSmallMProgram or DP4AMatMulNBitsProgram. And the same shape with 4 bits also fail on dp4. But the same shape can pass on non-dp4 path. The weird thing is that only [M=2 or3, N = 5120, K=3072] fails. If I set M = 1 or 4, the case pass. Or set N = 6144 or other value, the case can also pass. I haven't figured out the reason that why [M=2 or3, N = 5120, K=3072] are special for dp4.
But the CI GPU passes that case.

[daijh]

Thanks! I'll take a look now in case it blocks another PR of mine. :)

[daijh]

One date shows a minor deviation from the acceptable range. (0.105371 > 0.104557).
Suggest to increase current tolerance a bit.

Error: 4662: cur_expected -0.227832, cur_actual -0.122461, tolerance 0.104557