TensorRT backend not work when the device_id is greater than 0 in multiple threads.

[83204273]

Describe the issue

TensorRT backend work fine in multiple threads when the device_id is 0, but fails when the device_id is specified as a value greater than 0 and run in multiple threads.
The model is with dynamic shape.

2025-03-11 17:21:32.0860506 [E:onnxruntime:test, tensorrt_execution_provider.h:88 onnxruntime::TensorrtLogger::log] [2025-03-11 09:21:32 ERROR] IExecutionContext::enqueueV3: Error Code 1: CuTensor (Internal cuTensor permutate execute failed)
2025-03-11 17:21:32.1082442 [E:onnxruntime:test, tensorrt_execution_provider.h:88 onnxruntime::TensorrtLogger::log] [2025-03-11 09:21:32 ERROR] [checkMacros.cpp::nvinfer1::catchCudaError::212] Error Code 1: Cuda Runtime (invalid resource handle)
2025-03-11 17:21:32.1261594 [E:onnxruntime:, sequential_executor.cc:516 onnxruntime::ExecuteKernel] Non-zero status code returned while running TRTKernel_graph_main_graph_3446361497504689927_0 node. Name:'TensorrtExecutionProvider_TRTKernel_graph_main_graph_3446361497504689927_0_0' Status Message: TensorRT EP execution context enqueue failed.
Non-zero status code returned while running TRTKernel_graph_main_graph_3446361497504689927_0 node. Name:'TensorrtExecutionProvider_TRTKernel_graph_main_graph_3446361497504689927_0_0' Status Message: TensorRT EP execution context enqueue failed.

To reproduce

...

Urgency

No response

Platform

Windows

OS Version

Windows 11

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

onnxruntime-win-x64-gpu-1.21.0

ONNX Runtime API

C++

Architecture

X64

Execution Provider

TensorRT

Execution Provider Library Version

CUDA 12.3, TensorRT 10.9

[83204273]

the similar issus #19467

[83204273]

cudaSetDevice(mDevice);
add in each thread。

[chilo-ms]

Zhijie Zhu (@83204273)
We addressed this issue as well recently. #24010
Simply did the same thing, cudaSetDevice() for compute func that could be called by each thread.