Avoid repetitive creation of fp4/fp8 native-custom-op domains for NvTensorRtRtx EP - #27192
Conversation
There was a problem hiding this comment.
Pull request overview
This PR refactors the NvTensorRTRTX execution provider's custom ops initialization to improve efficiency and correctness. It introduces a flag-based approach to avoid redundant initialization of FP4/FP8 native custom ops and removes potentially dangerous manual deletion of statically-allocated objects.
Changes:
- Added
native_custom_ops_initializedflag to track whether native custom ops (TRT_FP4DynamicQuantize, TRT_FP8QuantizeLinear, TRT_FP8DequantizeLinear) have been created - Restructured control flow to add already-initialized native ops to the domain list without re-creating them
- Made release functions no-ops since the custom op domains are static objects managed by unique_ptr with static storage duration
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
304db7c to
3e87df1
Compare
I have synced the branch. |
|
Dmitri Smirnov (@yuslepukhin) Tianlei Wu (@tianleiwu) Can I get a review for this? |
|
/azp run Linux QNN CI Pipeline, Win_TRT_Minimal_CUDA_Test_CI,Windows ARM64 QNN CI Pipeline,Windows GPU Doc Gen CI Pipeline,Windows x64 QNN CI Pipeline |
|
Azure Pipelines successfully started running 4 pipeline(s). |
3e87df1 to
88bf125
Compare
Tianlei Wu (tianleiwu)
left a comment
Tianlei Wu (tianleiwu)
left a comment
There was a problem hiding this comment.
The changes look correct and robust.
Thread Safety: The use of static std::mutex effectively protects the initialization logic.
Resource Management: Removing the manual delete corresponds correctly to the static ownership model.
Logic Correctness: The separation of native_custom_ops_initialized check ensures that native ops are returned correctly even if custom_op_domain is empty, while preventing duplicate initialization.
The PR improves stability and prevents potential memory corruption.
|
Error in React Native CI Pipeline / React Native CI Android (pull_request) doesn't look related. Raw Logs - link
|
88bf125 to
3506183
Compare
|
/azp run Linux QNN CI Pipeline, Win_TRT_Minimal_CUDA_Test_CI, Windows ARM64 QNN CI Pipeline, Windows GPU Doc Gen CI Pipeline |
|
Azure Pipelines successfully started running 4 pipeline(s). |
Tianlei Wu (tianleiwu)
left a comment
Tianlei Wu (tianleiwu)
left a comment
There was a problem hiding this comment.
custom_op_domain->custom_ops_ could accumulate duplicates: The plugin registration path (the try/catch block that iterates getPluginCreatorList) doesn't check whether custom_op_domain already has ops — it relies on custom_op_domain->domain_ != "" as the guard. If that guard somehow fails (domain_ never set because the try block threw), on a subsequent call the plugins could be re-registered and created_custom_op_list / custom_op_domain->custom_ops_ would accumulate duplicates. This is a pre-existing issue not introduced by this PR, so it can be deferred.
The original TRT EP (tensorrt_execution_provider_custom_ops.cc) has the same double-free bug: It uses static unique_ptr but its ReleaseTensorRTCustomOpDomain still does delete ptr. This PR only fixes it for TRTRTX EP. This should be tracked as a follow-up.
Approve with minor suggestion: The code is correct, addresses all actionable review feedback, and the deferred architectural work is appropriately scoped for a follow-up. Consider filing a tracking issue for the broader ownership redesign (covering both TRT and TRTRTX EPs).
He is on vacation right now. Update the status to unblock the PR.
d5387d8
into
microsoft:main
Apply the same fix from NvTensorRtRtx EP (PR #27192) to the TRT EP: - ReleaseTensorRTCustomOpDomain: make no-op since objects are owned by static unique_ptrs in CreateTensorRTCustomOpDomainList() - ReleaseTensorRTCustomOpDomainList: just clear the vector instead of deleting static-owned objects - Add ownership documentation comments to static members Co-authored-by: tianleiwu <30328909+tianleiwu@users.noreply.github.com>
### Description Apply the same double-free fix from NvTensorRtRtx EP ([PR #27192](#27192)) to the TRT EP. `CreateTensorRTCustomOpDomainList()` owns domains/ops via static `unique_ptr`s, but `ReleaseTensorRTCustomOpDomain()` was manually `delete`-ing the same objects through raw pointers — double-free at program exit. - `ReleaseTensorRTCustomOpDomain()` → no-op (static `unique_ptr`s own the lifetime) - `ReleaseTensorRTCustomOpDomainList()` → `clear()` the reference vector only - Added ownership comments to static members matching NvTensorRtRtx EP style ### Motivation and Context PR #27192 review ([thread](#27192 (comment))) identified TRT EP has the identical bug pattern that was fixed in NvTensorRtRtx EP. The TRT EP code was the original source this pattern was borrowed from. @tianleiwu noted a follow-up PR was needed. <!-- START COPILOT CODING AGENT TIPS --> --- 🔒 GitHub Advanced Security automatically protects Copilot coding agent pull requests. You can protect all pull requests by enabling Advanced Security for your repositories. [Learn more about Advanced Security.](https://gh.io/cca-advanced-security) --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: tianleiwu <30328909+tianleiwu@users.noreply.github.com>
Description
Motivation and Context