[Web] Avoid unnecessary data copy for pre-allocated tensors

[Wanming Lin (Honry)]

Description

Ensure all pre-allocated tensors do not trigger unnecessary data copying. e.g. the WebNN EP always binds its tensor to 'ml-tensor'. In such cases, the tensor ID might change after binding, but copying data for these tensors should still be avoided.

Motivation and Context

This improves efficiency and avoids redundant operations.

[Wanming Lin (Honry)]

Yulong Wang (@fs-eire), PTAL, thanks!

[Guenther Schmuelling (guschmue)]

/azp run Linux QNN CI Pipeline,Win_TRT_Minimal_CUDA_Test_CI,Windows ARM64 QNN CI Pipeline,Windows GPU Doc Gen CI Pipeline,Windows x64 QNN CI Pipeline

[azure-pipelines]

Azure Pipelines successfully started running 5 pipeline(s).

[Yulong Wang (fs-eire)]

I may need to understand more details about why in this case, tensor is not equal to outputTensorHandles[i] but they are the same bound output. Is this a WebNN only case? In my understanding, if tensor !== outputTensorHandles[i], we should release tensor otherwise it's a memory leak.

[Wanming Lin (Honry)]

I may need to understand more details about why in this case, tensor is not equal to outputTensorHandles[i] but they are the same bound output. Is this a WebNN only case?

Currently this is WebNN only case, usually if we use pre-allocated output tensors, the ioBindingState will be null. After this PR, WebNN always bind its outputs to 'ml-tensor', therefore the ioBindingState is not null and then it will always call the wasm._OrtRunWithBinding() (which leads to the tensor !== outputTensorHandles[i])

In my understanding, if tensor !== outputTensorHandles[i], we should release tensor otherwise it's a memory leak.

So I need to release this tensor early?

[Yulong Wang (fs-eire)]

I may need to understand more details about why in this case, tensor is not equal to outputTensorHandles[i] but they are the same bound output. Is this a WebNN only case?

Currently this is WebNN only case, usually if we use pre-allocated output tensors, the ioBindingState will be null. After this PR, WebNN always bind its outputs to 'ml-tensor', therefore the ioBindingState is not null and then it will always call the wasm._OrtRunWithBinding() (which leads to the tensor !== outputTensorHandles[i])

In my understanding, if tensor !== outputTensorHandles[i], we should release tensor otherwise it's a memory leak.

So I need to release this tensor early?

For OrtRunWithBinding, the tensor should be the same to outputTensorHandles[i]:

onnxruntime/onnxruntime/wasm/api.cc

Lines 605 to 607 in 131cf40

	for (size_t i = 0; i < output_count; i++) {
	outputs[i] = binding_outputs[i];
	}

[Wanming Lin (Honry)]

I may need to understand more details about why in this case, tensor is not equal to outputTensorHandles[i] but they are the same bound output. Is this a WebNN only case?

Currently this is WebNN only case, usually if we use pre-allocated output tensors, the ioBindingState will be null. After this PR, WebNN always bind its outputs to 'ml-tensor', therefore the ioBindingState is not null and then it will always call the wasm._OrtRunWithBinding() (which leads to the tensor !== outputTensorHandles[i])

In my understanding, if tensor !== outputTensorHandles[i], we should release tensor otherwise it's a memory leak.

So I need to release this tensor early?

For OrtRunWithBinding, the tensor should be the same to outputTensorHandles[i]:

onnxruntime/onnxruntime/wasm/api.cc

Lines 605 to 607 in 131cf40

	for (size_t i = 0; i < output_count; i++) {
	outputs[i] = binding_outputs[i];
	}

Oh, that's odd, as I tested tensor !== outputTensorHandles[i] after OrtRunWithBinding, I will do more debug.

[Wanming Lin (Honry)]

Oh, that's odd, as I tested tensor !== outputTensorHandles[i] after OrtRunWithBinding, I will do more debug.

Yulong Wang (@fs-eire), after further debugging, I find there're two places change the address of the tensor.

1. wasm._OrtBindOutput -> BindOutputImpl

   onnxruntime/onnxruntime/core/session/IOBinding.cc

   Line 94 in e57dc2a

   output_names_.push_back(name);

2. wasm._OrtRunWithBinding -> GetBoundOutputValues
   

   onnxruntime/onnxruntime/core/session/onnxruntime_c_api.cc

   Line 1018 in e57dc2a

   value_dups.push_back(std::make_unique<OrtValue>(out_value));

[Wanming Lin (Honry)]

Yulong Wang (@fs-eire), friendly ping.

[Yulong Wang (fs-eire)]

Yulong Wang (@fs-eire), friendly ping.

please merge to latest main branch and add comments in line 866 "// TODO: revisit this part to ensure it works for WebGPU when both pre-allocated outputs and preferred location are specified"

[Wanming Lin (Honry)]

Yulong Wang (@fs-eire), friendly ping.

please merge to latest main branch and add comments in line 866 "// TODO: revisit this part to ensure it works for WebGPU when both pre-allocated outputs and preferred location are specified"

Yulong Wang (@fs-eire), done.

[Yulong Wang (fs-eire)]

/azp run Linux QNN CI Pipeline, Win_TRT_Minimal_CUDA_Test_CI, Windows ARM64 QNN CI Pipeline, Windows GPU Doc Gen CI Pipeline, Windows x64 QNN CI Pipeline

[azure-pipelines]

Azure Pipelines successfully started running 5 pipeline(s).

[Wanming Lin (Honry)]

Yulong Wang (@fs-eire), one unexpected failure CI, do I need to re-merge to latest main branch?