Ort openvino npu 1.17 master by sfatimar · Pull Request #19966 · microsoft/onnxruntime

sfatimar · 2024-03-18T15:19:09Z

Description

Add NPU to list of device supported.
Added changes for Support to OV 2024.0
Nuget packages removes packaging of OpenVINO DLL
Bug Fixes with Python API
Reverted Dockerfiles not being maintained.

Motivation and Context

NPU Device has been introduced by Intel in latest client systems
OpenVINO 2024.0 release is out.

…roved performance

…ecks for proxy plugins for NUM_STREAMS

… CompileModel stages

Since there's a common GlobalContext, when we create two Sessions in the same user code, the provider options of the first model gets overwritten with the second model's. To overcome this, this PR creates an instance of GlobalContext for each instance of OpenVINOExecutionProvider and uses it throughout wherever access to the GlobalContext's parameters are required.

… fallback code

…y the onnx frontend

sfatimar · 2024-03-20T17:35:15Z

@jywu-msft Updated the azure pipeline

George Wu (jywu-mysoft) · 2024-03-20T21:07:20Z

/azp run Linux OpenVINO CI Pipeline

azure-pipelines · 2024-03-20T21:07:32Z

Azure Pipelines successfully started running 1 pipeline(s).

sfatimar · 2024-03-21T03:49:50Z

Seems like one of unit test is giving accuracy mismatch. It worked fine in our systems. Checking on this

sfatimar · 2024-03-21T05:58:46Z

@jywu-msft Disabled the unit test , Can you please re run

George Wu (jywu-mysoft) · 2024-03-21T06:40:19Z

/azp run Linux OpenVINO CI Pipeline

azure-pipelines · 2024-03-21T06:40:31Z

Azure Pipelines successfully started running 1 pipeline(s).

Ignore Gelu tests in ONNX Backend test

sfatimar · 2024-03-21T13:46:27Z

The pipeline failed again :( . This time for Gelu Python Tests. One of the attributes approximate in Gelu is "none" in ONNX however we had defined it as "" string . Since it was part of latest ONNX Test suite and our onnx was not updated, this occurred. Hopefully it will pass this time.

George Wu (jywu-mysoft) · 2024-03-21T15:54:49Z

/azp run Linux OpenVINO CI Pipeline

azure-pipelines · 2024-03-21T15:55:01Z

Azure Pipelines successfully started running 1 pipeline(s).

George Wu (jywu-mysoft) · 2024-03-21T17:08:32Z

/azp run Windows GPU TensorRT CI Pipeline, onnxruntime-binary-size-checks-ci-pipeline, onnxruntime-python-checks-ci-pipeline, orttraining-linux-ci-pipeline, orttraining-linux-gpu-ci-pipeline, orttraining-ortmodule-distributed

George Wu (jywu-mysoft) · 2024-03-21T17:08:56Z

/azp run Linux CPU CI Pipeline, Linux CPU Minimal Build E2E CI Pipeline, Linux GPU CI Pipeline, Linux GPU TensorRT CI Pipeline, Linux Nuphar CI Pipeline, MacOS CI Pipeline, ONNX Runtime Web CI Pipeline, Windows CPU CI Pipeline, Windows GPU CI Pipeline

azure-pipelines · 2024-03-21T17:08:58Z

Azure Pipelines successfully started running 5 pipeline(s).

azure-pipelines · 2024-03-21T17:09:38Z

Azure Pipelines successfully started running 8 pipeline(s).

George Wu (jywu-mysoft) · 2024-03-21T17:10:26Z

/azp run Big Models, Linux QNN CI Pipeline, Windows ARM64 QNN CI Pipeline, Windows x64 QNN CI Pipeline

azure-pipelines · 2024-03-21T17:10:46Z

Azure Pipelines successfully started running 4 pipeline(s).

This reverts commit eab35c2.

Add NPU to list of device supported. Added changes for Support to OV 2024.0 Nuget packages removes packaging of OpenVINO DLL Bug Fixes with Python API Reverted Dockerfiles not being maintained. NPU Device has been introduced by Intel in latest client systems OpenVINO 2024.0 release is out. --------- Co-authored-by: Suryaprakash Shanmugam <suryaprakash.shanmugam@intel.com> Co-authored-by: Preetha Veeramalai <preetha.veeramalai@intel.com> Co-authored-by: Ubuntu <ubuntu@ubuntu-118727.iind.intel.com> Co-authored-by: hmamidix <hemax.sowjanya.mamidi@intel.com> Co-authored-by: vthaniel <vishnudas.thaniel.s@intel.com> Co-authored-by: saurabhkale17 <saurabh1.kale@intel.com>

### Description Add NPU to list of device supported. Added changes for Support to OV 2024.0 Nuget packages removes packaging of OpenVINO DLL Bug Fixes with Python API Reverted Dockerfiles not being maintained. ### Motivation and Context NPU Device has been introduced by Intel in latest client systems OpenVINO 2024.0 release is out. --------- Co-authored-by: Suryaprakash Shanmugam <suryaprakash.shanmugam@intel.com> Co-authored-by: Preetha Veeramalai <preetha.veeramalai@intel.com> Co-authored-by: Ubuntu <ubuntu@ubuntu-118727.iind.intel.com> Co-authored-by: hmamidix <hemax.sowjanya.mamidi@intel.com> Co-authored-by: vthaniel <vishnudas.thaniel.s@intel.com> Co-authored-by: saurabhkale17 <saurabh1.kale@intel.com>

Suryaprakash Shanmugam (sspintel) and others added 30 commits March 12, 2024 23:14

Add OpenVINO 2023.2 support

9e9c1fb

num_of_threads mapped to inference_num_threads property of CPU

f71ca3a

Use ONNX FrontEnd convert_model to facilitate external weights path

b656d06

Fix Lint issues

cce97c0

inference_num_threadas is applicable only for the CPU device

babc244

Fix Lint issues

b6e4af0

Add an user option to disable default dynamic model execution for imp…

6b8c867

…roved performance

Fix for runtime errors with dynamic shapes flag

4f7fe28

Fix conflict in gitignore

6704415

Revert removal of USE_OPENVINO macro in provider registration; Add ch…

359b71b

…ecks for proxy plugins for NUM_STREAMS

Enable OV CPU fallback for NPU compilation failures

fb6d981

Add NPU device in supported list of openvino devices

d2879b9

Handle dynamic shapes fallback for NPU to OV CPU

b7db23f

Remove NPU operator from static mapping

08f5b8c

Add NPU device; Revert num_of_threads to 1 to be default

d9fe4c1

Add support for LayerNormalization Op; NPU to go through ReadModel ->…

ff7070b

… CompileModel stages

Add device_precision access for UnsupportedOpModes

4121684

Fix an issue that shared global_context across subgraphs

e8fd60a

Remove static mapping of LayerNorm op for the NPU; Remove unused MLAS…

439afd3

… fallback code

Add support for UINT16 DTYPE in initializers, NPU, and CPU devices

593d870

Temporarily disable model domain check as it is yet to be supported b…

de8194c

…y the onnx frontend

Allow overriding NPU compiler type through an environmental variable

8fe5760

Remove deprecated model domain check

281ddf6

Remove unused parameter op_map

49d7f4a

Enable dynamic backend execution for NPU

0466f29

OV NPU fallback for OV CPU

e4cc1c9

The default should be false

da26118

Resetting num of threads to 0

6adbc90

Nuget does not need openvino windows dll

16c9a3e

Disable ActivationOpTest.Gelu for OpenVINO

9d72346

sfatimar mentioned this pull request Mar 21, 2024

Ort openvino npu 1.17 master intel/onnxruntime#351

Closed

Suryaprakash Shanmugam (sspintel) and others added 2 commits March 21, 2024 04:41

Ignore Gelu tests in ONNX Backend test

3eaa70a

Merge pull request #355 from intel/sp/ignore-gelu-tests-python

76d7e17

Ignore Gelu tests in ONNX Backend test

George Wu (jywu-mysoft) approved these changes Mar 22, 2024

View reviewed changes

George Wu (jywu-mysoft) merged commit eab35c2 into microsoft:main Mar 22, 2024

This was referenced Mar 24, 2024

[OpenVINO EP] Use OpenVINO cmake #19184

Closed

[Build] Find openvino based on env variable #19696

Closed

ONNX Runtime OpenVINO EP is way behind #19688

Closed

Yi Zhang (mszhanyi) pushed a commit that referenced this pull request Mar 26, 2024

Revert "Ort openvino npu 1.17 master (#19966)"

daa53a2

This reverts commit eab35c2.

This was referenced Apr 20, 2024

[onnxruntime] new ports for microsoft/onnxruntime v1.23.2 microsoft/vcpkg#36850

Merged

[onnxruntime] Update to v1.17.3 luncliff/vcpkg-registry#192

Merged

Uh oh!

Conversation

sfatimar commented Mar 18, 2024

Description

Motivation and Context

Uh oh!

sfatimar commented Mar 20, 2024

Uh oh!

George Wu (jywu-mysoft) commented Mar 20, 2024

Uh oh!

azure-pipelines Bot commented Mar 20, 2024

Uh oh!

sfatimar commented Mar 21, 2024

Uh oh!

sfatimar commented Mar 21, 2024

Uh oh!

George Wu (jywu-mysoft) commented Mar 21, 2024

Uh oh!

azure-pipelines Bot commented Mar 21, 2024

Uh oh!

sfatimar commented Mar 21, 2024

Uh oh!

George Wu (jywu-mysoft) commented Mar 21, 2024

Uh oh!

azure-pipelines Bot commented Mar 21, 2024

Uh oh!

George Wu (jywu-mysoft) commented Mar 21, 2024

Uh oh!

George Wu (jywu-mysoft) commented Mar 21, 2024

Uh oh!

azure-pipelines Bot commented Mar 21, 2024

Uh oh!

azure-pipelines Bot commented Mar 21, 2024

Uh oh!

George Wu (jywu-mysoft) commented Mar 21, 2024

Uh oh!

azure-pipelines Bot commented Mar 21, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants