CVS-175736 - [OVEP] Optimize Stateful Path: use output-to-input strategy to get the pairs of KV name by Kotomi-Du · Pull Request #845 · intel/onnxruntime

Kotomi-Du · 2025-11-06T01:56:36Z

Description

It is hardcoded for the path of identifying KV pairs. This PR is to make it general to fit most of the cases. It is a follow up of #821

Here is the strategy:

check output name with "present" to get output_tensor list
extract the pattern, e.g. key_***_%d; value_***_%d from output tesnor
apply extracted pattern for input name to get input_tensor list.

Jira Ticket :

https://jira.devtools.intel.com/browse/CVS-175736

Go to new ABI?

Yes

Copilot

Pull Request Overview

This PR optimizes the stateful path handling in OpenVINO by implementing a dynamic pattern extraction strategy to identify KV (key-value) pairs, replacing the previous hardcoded approach. The strategy extracts patterns from output tensors containing "present" and applies them to identify corresponding input tensors.

Key Changes:

Introduced regex-based pattern extraction from output tensor names to dynamically identify KV pairs
Added fallback mechanism using substring matching when pattern extraction yields no results
Refactored PatchStatefulDecoder to use the new pattern-based extraction functions

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

mklimenk · 2025-11-06T09:57:38Z

+      if (name.starts_with("present")) {
+        key_value_output_names.push_back(name);
+        std::smatch match;
+        if (std::regex_match(name, match, present_pattern)) {


Do we need a regular expression here? It seems like it can be simplified to a regular string manipulation: we already know that the string starts with "present" and we can find where the last underscore is, gathering the substring.
This is also a micro-optimization in terms of performance, since C++ regexps are compiled at runtime and are known for their bad performance.

RyanMetcalfeInt8 · 2025-11-07T02:00:11Z

+  auto [key_value_output_names, extracted_patterns] = ExtractKVPatternsFromOutputs(model);
+  auto [key_value_input_names, not_kv_inputs] = ExtractInputKVTensors(model, extracted_patterns);
+
+  std::cout << key_value_input_names.size() << ";" << key_value_output_names.size() << std::endl;


looks like a debug statement here.

RyanMetcalfeInt8 · 2025-11-07T02:02:51Z

  }

+  if (key_value_input_names.size() != key_value_output_names.size()) {
+    std::cout << "found different sizes btween key_value_input_names and key_value_output_names, they couldn't be paired" << std::endl;


I think that this one should be a runtime exception of some sort. I don't think we'd ever want to hit this state, return, and have the rest of the stateful flow continue on.

RyanMetcalfeInt8 · 2025-11-07T02:04:12Z

+
+  std::cout << key_value_input_names.size() << ";" << key_value_output_names.size() << std::endl;
  if (key_value_input_names.empty() || key_value_output_names.empty()) {
    std::cout << "no key_value_input_names or key_value_output_names found" << std::endl;


Same for here as below -- I think there should be a runtime exception thrown here. I don't think we'd ever intend for the stateful flow to get enabled, and not identify pairs of tensors to perform a make_stateful transformation on.

mklimenk · 2025-11-10T08:40:19Z

 // https://github.com/huggingface/optimum-intel/blob/main/optimum/exporters/openvino/stateful.py#L281
-void PatchStatefulDecoder(std::shared_ptr<ov::Model> model) {
+// Helper function to extract KV patterns from output names dynamically
+std::pair<std::vector<std::string>, std::vector<std::string>> ExtractKVPatternsFromOutputs(const std::shared_ptr<ov::Model>& model) {


This function returns two std::vectors only to check that the first one is non-empty and the second one is used as a sort of lookup table. Therefore, it can return std::optional<T> instead.

Each element in key_value_output_names will be passed as information for making stateful, it cannot be switched to std::optional.

It is updated to std::optional for patterns now.

mklimenk · 2025-11-10T08:41:01Z

-void PatchStatefulDecoder(std::shared_ptr<ov::Model> model) {
+// Helper function to extract KV patterns from output names dynamically
+std::pair<std::vector<std::string>, std::vector<std::string>> ExtractKVPatternsFromOutputs(const std::shared_ptr<ov::Model>& model) {
+  std::set<std::string> unique_patterns;


Consider switching to std::unordered_set<T> if you don't need the values to be sorted.

mklimenk · 2025-11-10T08:41:35Z

+      }
+    }
+  }
+  std::vector<std::string> extracted_patterns(unique_patterns.begin(), unique_patterns.end());


Is it necessary to construct a std::vector here? Would it be possible to return the set directly?

updated it to std::optional<std::pair<std::string, std::string>> now.

…terns have to be followed by _%d

Copilot

Pull Request Overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 5 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

RyanMetcalfeInt8

LGTM

RyanMetcalfeInt8 · 2025-11-13T19:33:44Z

@MayureshV1 -- can you please take a look and merge if you are okay with this?

Copilot

Pull Request Overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-11-14T21:52:16Z

+        size_t last_underscore_pos = name.rfind('_');
+        // Extract pattern between "present_" and the last underscore
+        if (last_underscore_pos != std::string::npos && last_underscore_pos > prefix_len) {
+          std::string pattern = name.substr(prefix_len, last_underscore_pos - prefix_len);


The pattern extraction assumes the format 'present__' but doesn't validate that the suffix after the last underscore is actually numeric. This could lead to incorrect pattern extraction if output names have different formats (e.g., 'present_key_cross_layer_0'). Consider validating the numeric suffix before extracting the pattern.

Copilot · 2025-11-14T21:52:17Z

+  if (unique_patterns.size() > 2) {
+    ORT_THROW("More than two unique KV patterns found in output names.");
+  }


The hardcoded limit of 2 patterns contradicts the PR's goal of making the code more general. Consider either removing this restriction or making it configurable, as models with different architectures might legitimately have more than two KV pattern types.

Suggested change

if (unique_patterns.size() > 2) {

ORT_THROW("More than two unique KV patterns found in output names.");

}

gblong1

LGTM

Kotomi-Du added 2 commits November 5, 2025 16:25

use output-to-input strategy to get the pairs of KV name

d44a556

minor change

1634684

Kotomi-Du requested review from MayureshV1 and RyanMetcalfeInt8 November 6, 2025 01:56

RyanMetcalfeInt8 reviewed Nov 6, 2025

View reviewed changes

Comment thread onnxruntime/core/providers/openvino/ov_stateful_patch_utils.cc Outdated

RyanMetcalfeInt8 reviewed Nov 6, 2025

View reviewed changes

Comment thread onnxruntime/core/providers/openvino/ov_stateful_patch_utils.cc

RyanMetcalfeInt8 requested a review from Copilot November 6, 2025 02:16

Copilot AI reviewed Nov 6, 2025

View reviewed changes

mklimenk reviewed Nov 6, 2025

View reviewed changes

remove regex for extracting pattern

f6b3c63

RyanMetcalfeInt8 reviewed Nov 7, 2025

View reviewed changes

Address review

165a7cd

mklimenk reviewed Nov 10, 2025

View reviewed changes

Kotomi-Du added 2 commits November 10, 2025 15:00

Design strict KV patterns: only two separately for key and value; pat…

8565e29

…terns have to be followed by _%d

simplify code structure

a873e7b

Kotomi-Du requested a review from Copilot November 12, 2025 18:44

Copilot AI reviewed Nov 12, 2025

View reviewed changes

address review

5d5d1f2

RyanMetcalfeInt8 reviewed Nov 12, 2025

View reviewed changes

Comment thread onnxruntime/core/providers/openvino/ov_stateful_patch_utils.cc

RyanMetcalfeInt8 reviewed Nov 12, 2025

View reviewed changes

Comment thread onnxruntime/core/providers/openvino/ov_stateful_patch_utils.cc Outdated

Kotomi-Du added 2 commits November 12, 2025 15:37

remove useless comment

cec822e

add brief example to explain the functionalities

38ae639

RyanMetcalfeInt8 approved these changes Nov 13, 2025

View reviewed changes

MayureshV1 requested review from Copilot and mklimenk November 14, 2025 21:51

Copilot AI reviewed Nov 14, 2025

View reviewed changes

gblong1 approved these changes Nov 14, 2025

View reviewed changes

Merge branch 'ovep-develop' into optimize_stateful_path

de5bc93

MayureshV1 merged commit d0bac3e into intel:ovep-develop Nov 15, 2025
3 of 5 checks passed

	if (unique_patterns.size() > 2) {
	ORT_THROW("More than two unique KV patterns found in output names.");
	}

Uh oh!

Conversation

Kotomi-Du commented Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Jira Ticket :

Go to new ABI?

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Kotomi-Du Nov 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

RyanMetcalfeInt8 left a comment

Choose a reason for hiding this comment

Uh oh!

RyanMetcalfeInt8 commented Nov 13, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Copilot AI Nov 14, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 14, 2025

Choose a reason for hiding this comment

Uh oh!

gblong1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Kotomi-Du commented Nov 6, 2025 •

edited

Loading

Kotomi-Du Nov 10, 2025 •

edited

Loading