Skip to content

Adds automatic loading of the external tensorrt_plugins library in the TensorRT-RTX Execution Provider to support custom transformer plugins #26558

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Chi Lo (chilo-ms) merged 1 commit into from
Nov 13, 2025
Merged

Adds automatic loading of the external tensorrt_plugins library in the TensorRT-RTX Execution Provider to support custom transformer plugins #26558

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions onnxruntime/core/providers/nv_tensorrt_rtx/nv_execution_provider_custom_ops.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,23 @@ common::Status CreateTensorRTCustomOpDomainList(std::vector<OrtCustomOpDomain*>&
is_loaded = true;
}

// Load external tensorrt_plugins library from EP directory (same location as nvinfer_plugin)
// This library contains GroupQueryAttention and RotaryEmbedding plugins for transformer models
try {
const auto& env = onnxruntime::GetDefaultEnv();
auto external_plugin_path = env.GetRuntimePath() +
PathString(LIBRARY_PREFIX ORT_TSTR("tensorrt_plugins") LIBRARY_EXTENSION);
void* external_plugin_handle = nullptr;
auto status = env.LoadDynamicLibrary(external_plugin_path, false, &external_plugin_handle);
if (status.IsOK()) {
LOGS_DEFAULT(INFO) << "[NvTensorRTRTX EP] External plugins loaded: tensorrt_plugins (GQA + RotaryEmbedding)";
} else {
LOGS_DEFAULT(VERBOSE) << "[NvTensorRTRTX EP] tensorrt_plugins library not found in runtime path (optional)";
}
} catch (const std::exception& e) {
LOGS_DEFAULT(VERBOSE) << "[NvTensorRTRTX EP] tensorrt_plugins library not available: " << e.what();
}

try {
// Get all registered TRT plugins from registry
LOGS_DEFAULT(VERBOSE) << "[NvTensorRTRTX EP] Getting all registered TRT plugins from TRT plugin registry ...";
Expand Down
Loading