AddExternalInitializers copies device (GPU) OrtValues per session instead of using them in place #29009

Describe the issue

SessionOptions::AddExternalInitializers (C API OrtApi::AddExternalInitializers) accepts a list of user-supplied OrtValues to use as initializers. When the supplied OrtValue is already on the EP device (e.g. a CUDA tensor) and the initializer is planned on that same device, ORT still allocates a fresh device tensor and copies into it during every session's initialization, instead of using the supplied device buffer in place.

This has two consequences:

1. Redundant device→device copy + extra allocation per session for every externally-supplied initializer.
2. No cross-session device-memory sharing. A common use case is sharing large weights (e.g. a token embedding table and lm_head) between two sessions that load separate ONNX graphs — for example a main decoder and an auxiliary multi-token-prediction / speculative-decode head. Supplying the same device OrtValue to both sessions via AddExternalInitializers does not collapse the duplicate on the GPU, because each session copies into its own buffer.

By contrast, OrtApi::AddInitializer (single-tensor) routes through initializers_to_share_map and does use the supplied OrtValue in place when its device matches the planned device (the use_user_supplied_initializer path), so it both avoids the copy and shares the buffer across sessions. AddExternalInitializers arguably should offer the same no-copy behavior for device tensors.

Root cause

AddExternalInitializers stores the values in SessionOptions::external_initializers. Graph::InjectExternalInitializedTensors turns them into in-memory external initializers, and for an owned-buffer tensor it keeps the user's OrtValue as-is:

if (user_tensor.OwnsBuffer()) {
  // If the user tensor has its own memory, we avoid copying
  tensor_proto = utils::TensorToTensorProto(user_tensor, name, use_tensor_buffer_true);
  ORT_ENFORCE(utils::HasExternalDataInMemory(tensor_proto), "Expecting this tensor_proto to have a pointer");
  ort_value = value;   // keeps the user's OrtValue (may be a device tensor)
}

That OrtValue is then retrievable via Graph::GetOrtValueInitializer. But in SaveInitializedTensors the graph-initializer branch only uses it directly when the planned device is CPU; for any non-CPU device it allocates a new tensor and copies, ignoring that the supplied value may already be on the planned device:

if (OrtValue ort_value_from_graph;
    graph.GetOrtValueInitializer(name, ort_value_from_graph)) {
  const auto& memory_info = (alloc != nullptr) ? alloc->Info() : memory_buffer->GetAllocInfo();
  if (memory_info.device == default_cpu_device) {
    // This is on CPU use directly from the graph
    ort_value = std::move(ort_value_from_graph);
  } else {
    // <-- always allocates a new device tensor + CopyTensorFromCPUToDevice,
    //     even when ort_value_from_graph already lives on memory_info.device
    Tensor tensor;
    ORT_RETURN_IF_ERROR(AllocateTensor(..., tensor, ...));
    ORT_RETURN_IF_ERROR(CopyTensorFromCPUToDevice(data_transfer_mgr,
                                                  ort_value_from_graph.Get<Tensor>(),
                                                  std::move(tensor), ort_value));
  }
}

Suggested fix

In the GetOrtValueInitializer branch, use the supplied OrtValue directly (no allocation, no copy) when its tensor already lives on the planned device — mirroring the existing CPU case and the AddInitializer sharing path:

const auto& graph_value_device = ort_value_from_graph.Get<Tensor>().Location().device;
if (memory_info.device == default_cpu_device || graph_value_device == memory_info.device) {
  // Planned on CPU, or the supplied initializer already lives on the planned device:
  // use it in place (no per-session allocation/copy; enables cross-session sharing).
  ort_value = std::move(ort_value_from_graph);
} else {
  // existing allocate-on-device + CopyTensorFromCPUToDevice fallback (cross-device)
}

This makes AddExternalInitializers with device OrtValues avoid the redundant copy and allows the same device buffer to be shared across multiple sessions, matching the AddInitializer behavior.

Urgency

Not urgent, but it blocks an efficient memory-sharing pattern for multi-graph models (e.g. speculative-decoding/MTP heads sharing the main model's embedding + lm_head on the GPU, ~2 GB for a large-vocab model).

Platform

Linux

OS Version

Ubuntu

ONNX Runtime Installation

Built from Source

ONNX Runtime Version or Commit ID

51dc139

Execution Provider

CUDA