Weightless support for all initializers - #29607
Merged
Merged
Conversation
Add API surface for expanding weightless mode to cover all initializers (internal and external), working in both JIT and AOT flows: Session options: - Add ep.enable_weightless: unified weightless option for all initializers - Add ep.context_source_model_path: runtime override for source model location - Deprecate ep.enable_weightless_ep_context_nodes (external-only, AOT-only) OrtEpApi: - Add GetWeightlessSupport: EP capability query for the handshake. App requests weightless (session option), EP confirms support (this API). OrtCompileApi: - Add ModelCompilationOptions_SetWeightlessCache: enable weightless for AOT compilation path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Clarify that extending OrtValue lifetime past Compile() for direct pointer caching is planned but not yet implemented. Currently, EPs should use drop_constant_initializers=false and access initializers via KernelContext_GetInput() at Compute() time. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… time Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… timing - ep.enable_weightless: clarify JIT uses KernelContext path (direct pointer caching is TODO), remove premature lifetime extension claim - ep.context_source_model_path: fix 'inference time' to 'session creation' Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add two new OrtApi functions for providing the source model's initializer data when creating a session from a weightless EPContext model: - SessionOptionsSetWeightlessSourceModelPath: file path to source model, acts as runtime override for the onnx_model_filename EPContext attribute. - SessionOptionsSetWeightlessSourceModelFromBuffer: in-memory byte buffer for scenarios where the source model is not on disk. Caller must keep the buffer alive for the session lifetime. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
SessionOptionsSetWeightlessSourceModelPath: - Stores file path in OrtSessionOptions for runtime override of onnx_model_filename EPContext attribute - Clears any previously set buffer source SessionOptionsSetWeightlessSourceModelFromBuffer: - Stores borrowed buffer pointer + size in OrtSessionOptions - Caller must keep buffer alive for session lifetime - Clears any previously set file path source ModelCompilationOptions_SetWeightlessCache: - Adds use_weightless_cache_ flag to ModelCompilationOptions - Sets ep.enable_weightless=1 in session config options - Registered in ort_compile_api table Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
When the app requests weightless mode and the EP confirms support via GetWeightlessSupport(), ORT automatically overrides drop_constant_initializers to false in GetCapability(). This ensures constant initializers remain as fused node inputs so the EP can access them via KernelContext_GetInput() at Compute() time. The override is logged at INFO level for diagnostics. Constructor queries the EP's weightless support at creation time and stores the result in weightless_enabled_ for use during GetCapability(). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add kOrtEpDevice_EpMetadataKey_WeightlessSupport to onnxruntime_ep_device_ep_metadata_keys.h. EPs set this during GetSupportedDevices() so apps can query device-specific weightless capability before compilation via EpDevice_EpMetadata(). Valid values: 'none', 'external_only', 'all'. Update all new weightless APIs from version 1.28 to 1.29: - Session options: ep.enable_weightless, ep.context_source_model_path - OrtEpApi: OrtWeightlessSupport enum, GetWeightlessSupport - OrtApi: SessionOptionsSetWeightlessSourceModelPath, SessionOptionsSetWeightlessSourceModelFromBuffer - OrtCompileApi: ModelCompilationOptions_SetWeightlessCache - Version gate in PluginExecutionProvider constructor Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Chi Lo (chilo-ms)
marked this pull request as ready for review
July 9, 2026 16:45
Javier Martinez (javier-intel)
approved these changes
Jul 10, 2026
Chi Lo (chilo-ms)
requested review from
Edward Chen (edgchen1) and
Jambay Kinley (jambayk)
July 27, 2026 18:00
Add two getter APIs to OrtEpApi so plugin EPs can retrieve the weightless source model during CreateEp/Compile: - SessionOptions_GetWeightlessSourceModelPath: returns the file path set by SessionOptionsSetWeightlessSourceModelPath, or NULL if not set. - SessionOptions_GetWeightlessSourceModelBuffer: returns the buffer and size set by SessionOptionsSetWeightlessSourceModelFromBuffer, or NULL/0 if not set. This completes the weightless source model plumbing: app sets the source model via OrtApi setters, EP retrieves it via OrtEpApi getters. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The weightless_enabled_ flag was set but never read after the drop_constant_initializers override was removed. The EP handles weightless behavior on its own via session options and GetWeightlessSupport(). ORT Core doesn't need to track it separately. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The source model file path is set via the session config string ep.context_source_model_path which the EP reads directly. No dedicated C API or OrtEpApi getter is needed for the file path case. Keep only the buffer-based APIs since byte buffers cannot be passed via config strings. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Documentation: - SessionOptionsSetWeightlessSourceModelFromBuffer: add note about kOrtSessionOptionEpContextSourceModelPath for file-based source models - kOrtSessionOptionEpContextSourceModelPath: mention SessionOptionsSetWeightlessSourceModelFromBuffer for buffer-based source Tests: - PluginEp_WeightlessAllInitializers_CompileApi: end-to-end test using ep.enable_weightless + ModelCompilationOptions_SetWeightlessCache - PluginEp_WeightlessSourceModelFromBuffer_Validation: C API validation tests for null pointer and zero-length buffer error cases Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…etWeightlessSourceModelBuffer doc Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This was referenced Aug 12, 2026
Closed
Open
This was referenced Aug 19, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Expand ORT's weightless mode to cover all initializers (internal and external), working in both JIT and AOT flows.
Problem
The existing ep.enable_weightless_ep_context_nodes session option is a string-based convention between the app and the EP with no ORT Core involvement. It only covers external initializers in the AOT flow, has no versioning or capability negotiation, and JIT flows cannot benefit.
Changes
Session options ( onnxruntime_session_options_config_keys.h ):
OrtEpApi ( onnxruntime_ep_c_api.h ):
OrtApi ( onnxruntime_c_api.h ):
OrtCompileApi ( onnxruntime_c_api.h ):
EpDevice metadata ( onnxruntime_ep_device_ep_metadata_keys.h ):
kOrtEpDevice_EpMetadataKey_WeightlessSupport: pre-compilation query via EpDevice_EpMetadata() . EP sets "weightless_support" = "none" / "external_only" / "all" during GetSupportedDevices() so apps can check device-specific capability before calling SetWeightlessCache() .Source model plumbing:
Future work
Motivation and Context