Cannot run TensorRT session on another thread on a device other than the default one (0)

[JulienTheron]

Describe the issue

Running a TensorRT session on device 1 works as long as you're using the same thread that created the session.
When running the session on another thread, the following errors occur:

2023-06-07 16:46:47.2457468 [E:onnxruntime:, tensorrt_execution_provider.h:73 onnxruntime::TensorrtLogger::log] [2023-06-07 20:46:47   ERROR] 1: [eltwise.cu::cuEltwise::dispatchOp::423] Error Code 1: Cuda Runtime (invalid resource handle)
2023-06-07 16:46:47.2492417 [E:onnxruntime:, sequential_executor.cc:514 onnxruntime::ExecuteKernel] Non-zero status code returned while running TRTKernel_graph_Model Name_5506887977754746303_0 node. Name:'TensorrtExecutionProvider_TRTKernel_graph_Model Name_5506887977754746303_0_0' Status Message: TensorRT EP execution context enqueue failed.

Using the CUDA EP instead works as expected, so this is something specific to the TensorRT EP.
This issue happens on ORT 1.14.1 and 1.15.0 (other versions untested).

To reproduce

Run this code on a machine which has at least 2 CUDA devices.
I included a simple model created in Python that adds two tensors.
Usage: model.onnx

#include <future>
#include <vector>

#include "onnxruntime_cxx_api.h"

using namespace Ort;
using namespace std;

int wmain(int argc, wchar_t* argv[])
{
  if (argc < 2)
    return -1;

  const int deviceId = 1;

  Env env;
  SessionOptions sessionOptions;
  OrtTensorRTProviderOptions tensorrtOptions = { .device_id = deviceId };
  sessionOptions.AppendExecutionProvider_TensorRT(tensorrtOptions);

  Session session(env, argv[1], sessionOptions);

  const vector<int64_t> shape { 1, 2, 2, 3 };

  vector inputBuffer1(12, 1.0f);
  vector inputBuffer2(12, 2.0f);

  auto memInfo = MemoryInfo::CreateCpu(OrtArenaAllocator, OrtMemTypeDefault);

  const char* inputNames[] = { "input1", "input2" };
  const Value inputValues[] =
  {
    Value::CreateTensor(memInfo, inputBuffer1.data(), inputBuffer1.size(), shape.data(), shape.size()),
    Value::CreateTensor(memInfo, inputBuffer2.data(), inputBuffer2.size(), shape.data(), shape.size())
  };

  const char* outputNames[] = { "output" };
  vector<float> outputBuffer(12);
  Value outputValues[] =
  {
    Value::CreateTensor(memInfo, outputBuffer.data(), outputBuffer.size(), shape.data(), shape.size())
  };

  auto fut = async(launch::async, [&]
  {
    session.Run(RunOptions(), inputNames, inputValues, size(inputNames), outputNames, outputValues, size(outputNames));
  });

  fut.get();

  return 0;
}

model.zip

Urgency

The workaround is to call cudaSetDevice(1) before session.Run. This is not ideal as the ONNX Runtime is supposed to hide these kind of implementation details to the user.

Platform

Windows

OS Version

22621.1702

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

Microsoft.ML.OnnxRuntime.Gpu 1.15.0

ONNX Runtime API

C++

Architecture

X64

Execution Provider

TensorRT

Execution Provider Library Version

TensorRT 8.6.1 CUDA 11.8 cuDNN 8.9.0

[chilo-ms]

JulienTheron
I don't have a Windows machine that has 2 Nvida GPUs, but from what you described the bug it's possible related to multithreading issue which has recently being fixed by the PR.

Could you build the ORT from main and have it a try? If not I probably can share the private build with you.
We are going to release ORT 1.16 at the end of August and hopefully can fix current issue you encountered.

[JulienTheron]

Thanks chilo-ms, I'll give it a try and get back to you.

[JulienTheron]

I confirm the issue is fixed in the main branch.

[chilo-ms]

Thanks for confirming!
BTW, this fix will be in the ORT 1.16 which is planning to release at the end of August.

[JulienTheron]

Fixed in 1.16.0.

[jax11235]

The bug is still there in 1.17.0, I tested in linux platform, and cuda backend can run fine, but tensorrt backend raises the same error as this issue.

[chilo-ms]

The bug is still there in 1.17.0, I tested in linux platform, and cuda backend can run fine, but tensorrt backend raises the same error as this issue.

Which language binding are you using? Python or C++?
Could you help check whether 1.16.0 has the issue?
We will check ORT 1.17.0 from our side.

[jax11235]

In C++, I tried 1.16.0,1.16.3,1.17.0, they all have this problem.
In Python, I tried 1.16.3, there are no problem.

[chilo-ms]

Thanks for raising this issue. We will try to repro and investigate this.

[jax11235]

I tried to build from source, and set to debug mode, but even debug mode can't get more useful information when these errors appears.I tried to set the environment variable CUDA_VISIBLE_DEVICES, but all threads can only use the same device id. I tried to use cudart to set device, but another error appeared. My only successful attempt is to use multiple processes and set CUDA_VISIBLE_DEVICES. Can you provide a workaround so that I can avoid using multiple processes?

[chilo-ms]

I tried to build from source, and set to debug mode, but even debug mode can't get more useful information when these errors appears.I tried to set the environment variable CUDA_VISIBLE_DEVICES, but all threads can only use the same device id. I tried to use cudart to set device, but another error appeared. My only successful attempt is to use multiple processes and set CUDA_VISIBLE_DEVICES. Can you provide a workaround so that I can avoid using multiple processes?

Current TRT EP only supports one device id per inference session, meaning all the threads performing InferenceSession.Run() are using the same device id just like you observed. And users can only change device id by provider option and can't change it using cudart to set devoce.

We will try to repro on our side.