[Build] MIGraphX CI completely gone for MIGraphX EP builds

[TedThemistokleous]

Describe the issue

All of the related MIGraphX Linux/Windows CI has been removed from your coverage.

This was caught as part of: #25516

This was the result of the changes found here: #25418

Subsequent discussions mentioned the need for an almalinux image to use. ROCm actively maintains this found here:

https://hub.docker.com/r/rocm/dev-almalinux-8/tags

Conda is not used here for this image if this is required as a starting point for security features.

Urgency

Extremely urgent. Mainline build breaks for MIGraphX EP during Windows side integration.

We require test coverage. Linux integration for AMD systems will be halted to main

Target platform

MI series cards

Build script

Internal build scripts. MigraphX CI is removed

Error / output

Builds work on windows but will break linux environments

Visual Studio Version

No response

GCC / Compiler Version

No response

[TedThemistokleous]

Changming Sun (@snnn) Nick Eubanks (@nieubank) if GPUs are an issue, we can arrange further GPUs for your environment and testing.

Please reach out if this is the issue. Let us know how we can mitigate things further and if you require additional pieces from AMD.

[TedThemistokleous]

cc Jeff Daily (@jeffdaily) - Something we should be raising as well. This effects our release and upstreaming of features/integration with windows side changes

[snnn]

I am not able to get any AMD GPU currently, because AMD GPUs are in high demand. I will try to raise the request again.

[snnn]

I applied for https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/gpu-accelerated/ndmi300xv5-series?tabs=sizebasic .
Please let me know if there are other choices.

[snnn]

And we lack instructions of how to setup ROCM windows build.

[TedThemistokleous]

Why do you need a windows build for Linux CI?

[TedThemistokleous]

Windows support should be here - https://rocm.docs.amd.com/projects/install-on-windows/en/latest/

Can we get the Linux CI up and running? Why was that removed?

I am not able to get any AMD GPU currently, because AMD GPUs are in high demand. I will try to raise the request again.

Right, but this CI was working until last week until support was pulled?

[TedThemistokleous]

Jeff Mendenhall (@jeffmend) is there anything further AMD needs so we can get CI back up and running?

I can't seem to find Meng Tangs handle to ping him as well. Looking to get a timeline and any other resources so we can get support back before introducing back in Windows builds.

[snnn]

Mainly, there are two things:

1. We do not have AMD GPU machines. The only SKU I found in Azure is an 8-GPUs MI300x that is for training purposes only. It's very popular. I cannot get it. Even if we got some, they will be very expensive and we cannot make a good use of them, since ORT's build pipelines only use one GPU a time. There might be some other SKUs could be work. But we will your help to confirm/verify.

2. The docker file we used for Linux build has several security and license issues. We need people to fix the issue. I have put the details in your last PR at [MIGraphx EP] Sync AMD changes upstream  #25338 (comment)

If we could get a Windows build work, we don't need to bother with docker. And I think right now the integration work is mainly for Windows.

[TedThemistokleous]

Right, but again, we still require the Linux CI build for Onnxruntime and AMD has a publicly available almalinux build hosted on dockerhub similar to the Nvidia cases you use. I still don't see why we've lost complete support for Linux builds here. we have all the components and this effects customers we're currently supporting.

[TedThemistokleous]

Mainly, there are two things:

1. We do not have AMD GPU machines. The only SKU I found in Azure is an 8-GPUs MI300x that is for training purposes only. It's very popular. I cannot get it. Even if we got some, they will be very expensive and we cannot make a good use of them, since ORT's build pipelines only use one GPU a time. There might be some other SKUs could be work. But we will your help to confirm/verify.

2. The docker file we used for Linux build has several security and license issues. We need people to fix the issue.  I have put the details in your last PR at [[MIGraphx EP] Sync AMD changes upstream  #25338 (comment)](https://github.com/microsoft/onnxruntime/pull/25338#issuecomment-3076634856)

If we could get a Windows build work, we don't need to bother with docker. And I think right now the integration work is mainly for Windows.

We require both windows and linux builds if we want to ensure proper integration as the linux environments were supported previously. What you're essentially saying by removing CI is that *Nix doesn't matter anymore in lieu of Windows which disregards previous releases.

[TedThemistokleous]

cc Scott McKay (@skottmckay)