Skip to content

[AIX] test-suites failure fixes - #25791

Merged
Tianlei Wu (tianleiwu) merged 21 commits into
microsoft:mainfrom
ranjitshs:aix-test-fix
Nov 6, 2025
Merged

Tianlei Wu (tianleiwu) merged 21 commits into
microsoft:mainfrom
ranjitshs:aix-test-fix

Conversation

@ranjitshs

@ranjitshs Ranjit Ranjan (ranjitshs) commented Aug 19, 2025 •

Copy link
Copy Markdown
Contributor

Description

This PR is to fix some of the test case failures mentioned in #25790

  1. cmake/onnxruntime_unittests.cmake
    To fix the onnxruntime_shared_lib_test crash
  2. include/onnxruntime/core/providers/utils/ort_graph_to_proto.h
    Added method for handling external data.
    For fixing , EpGraphTest.SerializeToProto_ConstantOfShape , EpGraphTest.SerializeToProto_Mnist, EpGraphTest.SerializeToProto_InputModelHasExternalIni
  3. onnxruntime/core/framework/tensorprotoutils.cc
    For fixing many test failures , related to float16 or Int16 type.
  4. onnxruntime/test/framework/endian_test.cc
    For fixing, ConvertRawDataInTensorProtoTest.FloatData and ConvertRawDataInTensorProtoTest.Int32Data

Motivation and Context

To fix the AIX (Big endian) related test failures.

@ranjitshs

Ranjit Ranjan (ranjitshs) commented Aug 19, 2025 •

Copy link
Copy Markdown
Contributor Author

Changming Sun (@snnn) Tianlei Wu (@tianleiwu) Dmitri Smirnov (@yuslepukhin)
FYI.
Could you please review this PR and let me know your inputs.

Also, please trigger CI workflows for this PR.

Comment thread onnxruntime/core/framework/tensorprotoutils.cc Outdated
@ranjitshs

Copy link
Copy Markdown
Contributor Author

Also, for handling external data I have added ConvertExternalData() in ort_graph_to_proto.h which will be called in case be BE system Before writing data to file.

With this change, all tests for onnxruntime_ep_graph_test are passing.

Comment thread onnxruntime/core/framework/tensorprotoutils.cc Outdated
Comment thread onnxruntime/core/framework/tensorprotoutils.cc Outdated
@ranjitshs

Copy link
Copy Markdown
Contributor Author

Dmitri Smirnov (@yuslepukhin)
Let me know if any further comment in this PR . Can we close this PR if no further comments.

I will spend time for rest failures and will raise separate PR.

1: [  FAILED  ] FlatbufferUtilsTest.ExternalWriteReadWithLoadInitializers
1: [  FAILED  ] InternalTestingEP.TestSaveAndLoadOrtModel
1: [  FAILED  ] OrtModelOnlyTests.ValidateOrtFormatModelDoesNotRunOptimizersInFullBuild

Comment thread onnxruntime/core/framework/tensorprotoutils.cc Outdated
Comment thread onnxruntime/core/framework/tensorprotoutils.cc Outdated
Comment thread onnxruntime/core/framework/tensorprotoutils.cc Outdated
Comment thread onnxruntime/core/framework/tensorprotoutils.cc Outdated
Comment thread onnxruntime/test/ep_graph/test_ep_graph.cc
Comment thread onnxruntime/test/ep_graph/test_ep_graph.cc Outdated
Comment thread include/onnxruntime/core/providers/utils/ort_graph_to_proto.h Outdated
@yuslepukhin

Copy link
Copy Markdown
Contributor

Added more comments, this also needs to be rebased.

@ranjitshs

Copy link
Copy Markdown
Contributor Author

Added more comments, this also needs to be rebased.

Will address these comments in next commit. Thanks for reviewing it.

@ranjitshs

Copy link
Copy Markdown
Contributor Author

I see in some OS Ci's is failing with below . I will change and make this method as API . Wondering why it's not hit in all OS.
Could be strict compiler flags or latest compiler is used where it's failing.

/mnt/vss/_work/onnxruntime/onnxruntime/include/onnxruntime/core/providers/utils/ort_graph_to_proto.h:720:20: error: 'static' function 'ConvertExternalData' declared in header file should be declared 'static inline' [-Werror,-Wunneeded-internal-declaration]
  720 | static Ort::Status ConvertExternalData(const OrtValueInfo* value_info, void* data, size_t bytes) {

Comment thread include/onnxruntime/core/providers/utils/ort_graph_to_proto.h Outdated
Comment thread include/onnxruntime/core/providers/utils/ort_graph_to_proto.h Outdated
@ranjitshs

Copy link
Copy Markdown
Contributor Author

Dmitri Smirnov (@yuslepukhin) Adrian Lizarraga (@adrianlizarraga)
To not put any dependency on ort_graph_to_proto.h, I am thinking to add the required BE related method in this header file as static method . I am testing my changes and will upload for review.

@ranjitshs

Copy link
Copy Markdown
Contributor Author

Dmitri Smirnov (@yuslepukhin)
I was going through CI failures . There are some python related import failures, not related to this PR.

In windows debug X64, I see below . But same is passing in release mode.
I don't have windows for analysing it. Any suggestion .
I am thinking to put from line num 762 to 775 under if !defined(_WIN32)

Error: E:\_work\onnxruntime\onnxruntime\include\onnxruntime\core\providers\utils\ort_graph_to_proto.h(762,1): error C2220: the following warning is treated as an error [E:\_work\onnxruntime\onnxruntime\build\Debug\onnxruntime_ep_graph_test.vcxproj]
Warning: E:\_work\onnxruntime\onnxruntime\include\onnxruntime\core\providers\utils\ort_graph_to_proto.h(762,1): warning C4702: unreachable code [E:\_work\onnxruntime\onnxruntime\build\Debug\onnxruntime_ep_graph_test.vcxproj]
Warning: E:\_work\onnxruntime\onnxruntime\include\onnxruntime\core\providers\utils\ort_graph_to_proto.h(768,1): warning C4702: unreachable code [E:\_work\onnxruntime\onnxruntime\build\Debug\onnxruntime_ep_graph_test.vcxproj]
Warning: E:\_work\onnxruntime\onnxruntime\include\onnxruntime\core\providers\utils\ort_graph_to_proto.h(771,1): warning C4702: unreachable code [E:\_work\onnxruntime\onnxruntime\build\Debug\onnxruntime_ep_graph_test.vcxproj]
Warning: E:\_work\onnxruntime\onnxruntime\include\onnxruntime\core\providers\utils\ort_graph_to_proto.h(773,1): warning C4702: unreachable code [E:\_work\onnxruntime\onnxruntime\build\Debug\onnxruntime_ep_graph_test.vcxproj]
Warning: E:\_work\onnxruntime\onnxruntime\include\onnxruntime\core\providers\utils\ort_graph_to_proto.h(775,1): warning C4702: unreachable code [E:\_work\onnxruntime\onnxruntime\build\Debug\onnxruntime_ep_graph_test.vcxproj]

@ranjitshs

Copy link
Copy Markdown
Contributor Author

Hi Dmitri Smirnov (@yuslepukhin)
I have updated the code to fix windows debug build warning mentioned above.
Could you please run CI workflow .

@ranjitshs

Copy link
Copy Markdown
Contributor Author

Dmitri Smirnov (@yuslepukhin)
I see some new wanring(converted to error) under mainly WIN workflows.

Error: D:\a\_work\onnxruntime\onnxruntime\include\onnxruntime\core\providers\utils\ort_graph_to_proto.h(758,84): error C2220: the following warning is treated as an error [D:\a\_work\onnxruntime\onnxruntime\RelWithDebInfo\onnxruntime_ep_graph_test.vcxproj] 
Warning: D:\a\_work\onnxruntime\onnxruntime\include\onnxruntime\core\providers\utils\ort_graph_to_proto.h(758,84): warning C4100: 'bytes': unreferenced formal parameter [D:\a\_work\onnxruntime\onnxruntime\RelWithDebInfo\onnxruntime_ep_graph_test.vcxproj] 
Warning: D:\a\_work\onnxruntime\onnxruntime\include\onnxruntime\core\providers\utils\ort_graph_to_proto.h(758,71): warning C4100: 'data': unreferenced formal parameter [D:\a\_work\onnxruntime\onnxruntime\RelWithDebInfo\onnxruntime_ep_graph_test.vcxproj] 
Warning: D:\a\_work\onnxruntime\onnxruntime\include\onnxruntime\core\providers\utils\ort_graph_to_proto.h(758,53): warning C4100: 'value_info': unreferenced formal parameter [D:\a\_work\onnxruntime\onnxruntime\RelWithDebInfo\onnxruntime_ep_graph_test.vcxproj]

Update the code to handle unreferenced function parameters
Once you get time, please trigger CI once. Hope, this time it will go through. sorry for inconvenience

@yuslepukhin

Copy link
Copy Markdown
Contributor

You can comment out the names of the unreferenced params

@ranjitshs

Copy link
Copy Markdown
Contributor Author

Dmitri Smirnov (@yuslepukhin)
I see 4 pending checks . What's needs to be done for that.

@yuslepukhin

Copy link
Copy Markdown
Contributor

/azp run Linux QNN CI Pipeline, Win_TRT_Minimal_CUDA_Test_CI,Windows ARM64 QNN CI Pipeline,Windows GPU Doc Gen CI Pipeline,Windows x64 QNN CI Pipeline

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 4 pipeline(s).

@ranjitshs

Copy link
Copy Markdown
Contributor Author

Dmitri Smirnov (@yuslepukhin)
One of check Windows ARM64 QNN CI Pipeline above is timing out .. I don't see any build issues in log.
Please re-trigger it.

@ranjitshs

Copy link
Copy Markdown
Contributor Author

Dmitri Smirnov (@yuslepukhin)
If we are okay with these changes , then please approve and merge.

@tianleiwu

Copy link
Copy Markdown
Contributor

/azp run Windows ARM64 QNN CI Pipeline

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@ranjitshs

Copy link
Copy Markdown
Contributor Author

Hi Dmitri Smirnov (@yuslepukhin) Tianlei Wu (@tianleiwu)
BUILD_QNN_EP STATIC_LIB is failed with below error. Please re-trigger this workflow.

Starting: NuGetToolInstaller
==============================================================================
Task         : NuGet tool installer
Description  : Acquires a specific version of NuGet from the internet or the tools cache and adds it to the PATH. Use this task to change the version of NuGet used in the NuGet tasks.
Version      : 1.260.0
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/tool/nuget
==============================================================================
You are using a query match on the version string. Behavior changes or breaking changes might occur as NuGet updates to a new version.
ERR:write EPROTO C81D0000:error:0A000438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error:c:\ws\deps\openssl\openssl\ssl\record\rec_layer_s3.c:1605:SSL alert number 80

Finishing: NuGetToolInstaller

Comment thread include/onnxruntime/core/providers/utils/ort_graph_to_proto.h Outdated
@ranjitshs

Copy link
Copy Markdown
Contributor Author

Dmitri Smirnov (@yuslepukhin)
Update the code as per your suggestion. Please check once and trigger workflows

@tianleiwu

Copy link
Copy Markdown
Contributor

/azp run Linux QNN CI Pipeline, Win_TRT_Minimal_CUDA_Test_CI,Windows ARM64 QNN CI Pipeline,Windows GPU Doc Gen CI Pipeline,Windows x64 QNN CI Pipeline

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 4 pipeline(s).

@tianleiwu
Tianlei Wu (tianleiwu) enabled auto-merge (squash) November 6, 2025 01:28
@tianleiwu
Tianlei Wu (tianleiwu) merged commit 760eea4 into microsoft:main Nov 6, 2025
168 of 185 checks passed
@ranjitshs

Copy link
Copy Markdown
Contributor Author

Dmitri Smirnov (@yuslepukhin) Tianlei Wu (@tianleiwu)
Thanks for reviewing/merging this PR.

@ranjitshs
Ranjit Ranjan (ranjitshs) deleted the aix-test-fix branch November 6, 2025 14:27
Rohanjames1997 pushed a commit to Rohanjames1997/onnxruntime that referenced this pull request Dec 4, 2025
### Description
This PR is to fix some of the test case failures mentioned in
microsoft#25790

1. cmake/onnxruntime_unittests.cmake
   To fix the onnxruntime_shared_lib_test crash
2. include/onnxruntime/core/providers/utils/ort_graph_to_proto.h
   Added method for handling external data.
For fixing , EpGraphTest.SerializeToProto_ConstantOfShape ,
EpGraphTest.SerializeToProto_Mnist,
EpGraphTest.SerializeToProto_InputModelHasExternalIni
3. onnxruntime/core/framework/tensorprotoutils.cc
   For fixing many test failures , related to float16 or Int16 type.
4. onnxruntime/test/framework/endian_test.cc
For fixing, ConvertRawDataInTensorProtoTest.FloatData and
ConvertRawDataInTensorProtoTest.Int32Data
  
### Motivation and Context
To fix the AIX (Big endian) related test failures.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants