Sync with Microsoft ONNX Runtime - 22082026 - #1266
Closed
ai-fw-intg wants to merge 28 commits into
Closed
Conversation
…osoft#32053) ### Description Add a new EP allowlist that includes WebGPU (`cpu_acl_cuda_dml_webgpu_eps`), and use it for the Level-2 `GeluFusion` and `BiasGeluFusion` transformers. This allows the erf-based GELU pattern (`Div` -> `Erf` -> `Add` -> `Mul` -> `Mul`, optionally preceded by a bias Add) to fuse into the WebGPU EP's kMSDomain Gelu and BiasGelu kernels instead of running as separate elementwise dispatches. This covers models using older opsets that the opset >= 20 Level-1 `GeluFusion` transformer does not handle. ### Performance Impact In native WebGPU build, the vision submodel of zero-shot image classification achieved a 1.11x wall-clock speedup on Panther Lake and a 1.15x speedup on Wildcat Lake. | Platform | Latency reduction | Speedup | |--------------------------|-------------------|---------| | Intel Wildcat Lake (WCL) | −13.0% | 1.15× | | Intel Panther Lake (PTL) | −9.9% | 1.11× | This PR addresses the `Gelu` and `BiasGelu` items listed in [microsoft#29841](microsoft#29841). --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Upgrade cutlass from 4.4.2 to 4.7 Upgrade cudnn-frontend from 1.24 to 1.27
### Description <!-- Describe your changes. --> - Use commit timestamps for plugin EP dev versions - Update set_plugin_ep_build_variables.py to use argparse ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Python dev package versions were using only the commit date. This prevented packages from multiple commits on the same day from being published. Using a finer granularity (from day to seconds) should mitigate it. It's probably good enough if we only expect to publish from `main` and release branches. --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
To avoid OOM in cuda 13 plugin ep build: ``` ##[error]The agent worker exited with code 137, which means it ran out of memory. Make sure the agent (container) host has sufficient memory configured. ```
…t#32057) Widen each 8-lane int8 product group into the int32 accumulators before multiplying the second half of the packed K block. This avoids the signed int16 overflow that occurs when two `-128 * -128` products share a halfword lane. ### Description * Update the plain-NEON ARM64/AArch64 SymmQgemm S8 kernels to reduce each 8-lane product group into int32 before the next multiply. * Keep the ARM64 and AArch64 assembly implementations in sync. * Remove the temporary non-dotprod test guard introduced in microsoft#31606 so the existing signed-input regression coverage runs on the plain-NEON path again. ### Motivation and Context Fixes microsoft#31573. The previous `smull` + `smlal` sequence accumulated two int8 products in a signed int16 lane before widening. For the extreme case, `(-128 * -128) + (-128 * -128) = 32768`, which overflows int16. The revised sequence reduces each product group into the int32 accumulators before processing the second half of the packed K block. ### Validation The existing signed-input regression test is re-enabled for non-dotprod ARM64. The repository also contains `onnxruntime_mlas_benchmark` with `SYMMQGEMM/SignedActivation`; no representative non-dotprod Arm64 hardware was available for a trustworthy throughput comparison, so no performance numbers are claimed here.
This pull request addresses edge cases in the `TensorScatter` operator, particularly around handling large or potentially overflowing `write_indices` values. It also improves test coverage for these scenarios, ensuring robust and correct behavior in both linear and circular modes. ### Bug fixes and safety improvements * Fixed a potential overflow bug in linear mode by updating the bounds check to prevent `write_indices` values that could cause overflow when added to `sequence_length`. (`onnxruntime/core/providers/cpu/llm/tensorscatter.cc`) * Refined the circular mode logic to correctly handle very large `write_indices` values, ensuring correct wraparound behavior without overflow. (`onnxruntime/core/providers/cpu/llm/tensorscatter.cc`) ### Test coverage enhancements * Added a test for linear mode that verifies the operator fails gracefully when `write_indices` addition would overflow, ensuring the new bounds check is enforced. (`onnxruntime/test/providers/cpu/llm/tensorscatter_op_test.cc`) * Added a test for circular mode to confirm that very large `write_indices` values wrap correctly without overflow, verifying correct wraparound logic. (`onnxruntime/test/providers/cpu/llm/tensorscatter_op_test.cc`) * Included `<limits>` header to support the use of `std::numeric_limits<int64_t>::max()` in tests. (`onnxruntime/test/providers/cpu/llm/tensorscatter_op_test.cc`)
This pull request improves the robustness and correctness of handling external data references in ONNX Runtime, particularly focusing on rejecting invalid or ambiguous "location" entries and ensuring in-memory reference tags are not treated as file paths. The changes also enhance path validation logic and add new unit tests to cover these scenarios. **Validation and error handling improvements:** - Added a check in `ExternalDataInfo::Create` to reject duplicate "location" entries in the external data info, ensuring only one "location" is allowed per tensor. [[1]](diffhunk://#diff-df6326d7af04c52e54c61249ff8b1980da9bac81192ee2e63c5b2280d7b91e05R32) [[2]](diffhunk://#diff-df6326d7af04c52e54c61249ff8b1980da9bac81192ee2e63c5b2280d7b91e05L42-R47) - Updated path validation logic in `ValidateExternalDataPathFromDir` and `ValidateExternalDataPath` to explicitly reject in-memory reference tags (used for internal memory mapping) as invalid file paths, alongside empty and absolute paths. [[1]](diffhunk://#diff-d31e9fbe0f5334fcd949833e035f2b25d5ae810dcd505c545f6b372b546b1406L410-R417) [[2]](diffhunk://#diff-d31e9fbe0f5334fcd949833e035f2b25d5ae810dcd505c545f6b372b546b1406L423-R437) [[3]](diffhunk://#diff-d31e9fbe0f5334fcd949833e035f2b25d5ae810dcd505c545f6b372b546b1406L438-R446) [[4]](diffhunk://#diff-d31e9fbe0f5334fcd949833e035f2b25d5ae810dcd505c545f6b372b546b1406L448-R457) [[5]](diffhunk://#diff-d31e9fbe0f5334fcd949833e035f2b25d5ae810dcd505c545f6b372b546b1406L469-R477) [[6]](diffhunk://#diff-d31e9fbe0f5334fcd949833e035f2b25d5ae810dcd505c545f6b372b546b1406L497-R509) **Functionality and code clarity:** - Refined `HasExternalDataInMemory` to correctly detect in-memory references even if there are multiple "location" entries, improving reliability. **Testing enhancements:** - Added new unit tests to verify that duplicate "location" entries are rejected and that in-memory reference tags are correctly handled and rejected by validation logic. [[1]](diffhunk://#diff-d75ec5db9cc4642f78b6ff568aff6d10398fc211b0fb7c862d3ec88738e3eda6R146-R188) [[2]](diffhunk://#diff-d75ec5db9cc4642f78b6ff568aff6d10398fc211b0fb7c862d3ec88738e3eda6R771-R782) These changes collectively strengthen the validation of external data references, prevent ambiguous or invalid configurations, and ensure that in-memory tags are never misinterpreted as file paths.
This pull request improves the robustness and correctness of the `scale_by_axis` method in the `Initializer` class and adds new tests to ensure proper handling of edge cases, especially with empty tensors and invalid scaler inputs. **Improvements to `scale_by_axis` implementation:** * Updated the calculation of `block_size` and `num_blocks` in the `scale_by_axis` method of the `Initializer` class to use more precise and consistent dimension handling, addressing potential bugs with axis indexing. (`onnxruntime/core/optimizer/initializer.cc`) **Testing enhancements:** * Added a new test case (`ScaleByAxisEmptyTensor`) to cover scenarios where the target or scaler tensors are empty, ensuring that the method does not throw unexpectedly and returns the correct size. The test also checks for proper exception handling with invalid scaler sizes and data types. (`onnxruntime/test/optimizer/initializer_test.cc`) * Included the `span_utils.h` header in the test file to support the new test cases using spans. (`onnxruntime/test/optimizer/initializer_test.cc`)
This pull request updates the logic for determining whether a node's input is a valid initializer for fusion and adds a new unit test to ensure that initializers which can be overridden (i.e., initializers that are also graph inputs) are properly skipped during fusion. The main changes are: ### Logic Update * In `IsNodeValidForFusion`, the check for a valid initializer now uses `graph_utils::GetConstantInitializer`, ensuring that only constant initializers (not those that can be overridden by graph inputs) are considered valid for fusion. ### Testing Improvements * Added a new test, `FuseInitializersSkipsOverridableInitializer`, to verify that the transformer skips fusion for initializers that are also graph inputs and can be overridden. This test ensures the fusion transformation does not incorrectly modify such initializers. * Included the `graph_transform_test_builder.h` header to support the new test.
This pull request improves input validation for the `Normalizer` operator by adding a check to reject scalar (rank-0) inputs and introducing a corresponding unit test to ensure this behavior. The main changes are as follows: **Input validation improvements:** * Added a check in `normalizer.cc` to return an error if the input tensor has rank 0, ensuring only rank 1 or 2 inputs are accepted. **Testing enhancements:** * Added a new test case in `normalizer_test.cc` (`ScalarInputRejected`) to verify that scalar inputs are correctly rejected with an appropriate error message.
This pull request strengthens input validation for the Conv operator by adding explicit checks for the bias tensor shape and size, and adds a new unit test to verify this behavior. The main changes are as follows: **Input validation improvements:** * Added a check in `Conv<T>::Compute` (and the float specialization) to ensure that the bias tensor `B`, if provided, is a 1D tensor whose size matches the number of output channels (`M`). If not, an informative error message is returned. [[1]](diffhunk://#diff-17bd0d956bdc88650a3b0fc11efe67ee921f0e2f0d5e5ab3e73925cd4bb510d7R75-R77) [[2]](diffhunk://#diff-17bd0d956bdc88650a3b0fc11efe67ee921f0e2f0d5e5ab3e73925cd4bb510d7R261-R263) **Testing enhancements:** * Introduced a new test case `Conv2D_InvalidBiasSize` in `conv_op_test.cc` to verify that the Conv operator correctly fails when the bias tensor does not have the expected size, ensuring the new validation logic works as intended.
This pull request improves the handling and validation of empty set reductions in the ONNX Runtime CPU reduction operators. It enforces stricter checks for the axes input, prevents unnecessary memory operations, and adds comprehensive tests to verify correct behavior for various edge cases. **Validation and Error Handling Improvements:** - Enforces that the `axes` tensor input must be a 1D vector and only processes it if it is present, improving robustness and error messages for invalid input shapes. - Adds a check to ensure that memory copying only occurs when the input tensor is non-empty, avoiding redundant operations. **Logic and Flow Adjustments:** - Refactors the order of reduction logic to first handle empty axes cases before checking for empty set input, ensuring correct execution flow and output. [[1]](diffhunk://#diff-61c3aeab1bffa9fc6cac3ace83df3a778d14c8f8d02a53ab46e5b776e567cdd4L971-R986) [[2]](diffhunk://#diff-61c3aeab1bffa9fc6cac3ace83df3a778d14c8f8d02a53ab46e5b776e567cdd4L1016-R1031) **Testing Enhancements:** - Adds new tests to verify: - Reduction when the optional `axes` input is missing (should reduce all dimensions). - Validation that the `axes` tensor must be a vector, with expected error messages for invalid cases. - Correct behavior for the `noop_with_empty_axes` attribute, both when axes are omitted and when an empty axes tensor is provided.
This pull request enhances the validation logic for segment embedding inputs in the quantized `QEmbedLayerNorm` operator and significantly improves test coverage to ensure robustness against partial or inconsistent segment embedding input scenarios. **Validation Logic Improvements:** - Enforces that all segment embedding-related inputs (`segment_ids`, `segment_embedding`, `segment_embedding_scale`, `segment_embedding_zero_point`) must be provided together or all omitted, preventing invalid partial configurations. [[1]](diffhunk://#diff-ba9db626fd260b1ca5d0794268c616200df2929f2cf66cf67c383634a3e66727R190-R191) [[2]](diffhunk://#diff-ba9db626fd260b1ca5d0794268c616200df2929f2cf66cf67c383634a3e66727R203-L202) **Test Suite Enhancements:** - Refactors the test harness (`RunTest`) to allow fine-grained control over which segment embedding inputs are present, using a bitmask for flexible test scenarios. - Updates test input logic to conditionally add each segment embedding input based on the bitmask, enabling the simulation of all possible partial input combinations. [[1]](diffhunk://#diff-0396e3396c5f66dd6e1d6daac33d667ee873c93bc750b09650a9409af8df9a0aL67-R72) [[2]](diffhunk://#diff-0396e3396c5f66dd6e1d6daac33d667ee873c93bc750b09650a9409af8df9a0aL82-R87) [[3]](diffhunk://#diff-0396e3396c5f66dd6e1d6daac33d667ee873c93bc750b09650a9409af8df9a0aL114-R119) [[4]](diffhunk://#diff-0396e3396c5f66dd6e1d6daac33d667ee873c93bc750b09650a9409af8df9a0aL140-R145) - Adds a new test, `PartialSegmentInputsRejected`, which systematically verifies that any partial provision of segment embedding inputs is correctly rejected by the operator, aligning with the new validation logic. - Modifies the test execution to expect failures with a clear error message when partial segment embedding inputs are provided.
This pull request improves input validation for the `ScatterND` operator and adds a corresponding unit test to ensure that invalid input is properly rejected. Input validation: * Added a check in `scatter_nd.h` to ensure that the last dimension of the `indices` tensor is at least 1, returning an error if it is not. Testing: * Added a unit test in `scatter_nd_op_test.cc` (`ScatterND_rejects_zero_index_depth`) to verify that the operator rejects `indices` tensors with a zero-sized last dimension.
This pull request updates the ONNX Runtime Python bindings to improve memory management and API flexibility for asynchronous inference sessions. The main changes ensure that Python objects used as inputs to `run_async` are properly referenced and not prematurely garbage collected, and that the API is more Pythonic by accepting `py::object` for session and run options. It also adds a test to verify that input arrays are kept alive during asynchronous execution. **Improvements to memory management and API flexibility:** * The `AsyncResource` struct now stores references to Python feed objects, the session, and run options to ensure they are kept alive for the duration of async execution (`onnxruntime/python/onnxruntime_pybind_state.cc`). * The `run_async` binding now accepts `py::object` for the session and run options, and internally casts them as needed. This makes the API more Pythonic and flexible (`onnxruntime/python/onnxruntime_pybind_state.cc`). * When preparing feeds for `run_async`, the code now stores each input object in `feed_objects` before creating the corresponding `OrtValue`, ensuring the Python objects are not garbage collected too early (`onnxruntime/python/onnxruntime_pybind_state.cc`). **Testing and validation:** * A test using `weakref` was added to verify that input arrays passed to `run_async` remain alive until the callback is invoked, preventing premature garbage collection (`onnxruntime/test/python/onnxruntime_test_python.py`) [[1]](diffhunk://#diff-bc2d3954a8ed883e3036dc7a675c7ba13b6299e1079728e7b6069cab1f49479cR690) [[2]](diffhunk://#diff-bc2d3954a8ed883e3036dc7a675c7ba13b6299e1079728e7b6069cab1f49479cR702) [[3]](diffhunk://#diff-bc2d3954a8ed883e3036dc7a675c7ba13b6299e1079728e7b6069cab1f49479cL712-R721). * The test also exercises explicit deletion and garbage collection of inputs, session, and run options to ensure robustness (`onnxruntime/test/python/onnxruntime_test_python.py`).
…OOB read (microsoft#29461) ### Description The Split operator's split-as-attribute path validates that each split size is non-negative (in the constructor), but the split-as-input-tensor path skips that check and only validates the aggregate (sum == axis dim, count == num_outputs). A crafted negative split size like [6, -2] on an axis of size 4 passes the aggregate check ( 6 + (-2) = 4 ) and causes the kernel to copy 6 rows from a 4-row input — an out-of-bounds read. Changes: • split.h ( PrepareForCompute ): Per-element >= 0 validation. Covers CPU, WebGPU, shared-provider paths. • cuda/tensor/split.cc ( PrepareForComputeLocal ): Same fix in the CUDA copy. • split_op_test.cc : NegativeSplitSizeInputTensor test with split = [6, -2] expecting failure. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1d814472-441f-441d-bd46-931956efc1cd
### Description <!-- Describe your changes. --> Reject negative and out-of-range border and scale attributes without overflow. Align DirectML validation, handle empty CUDA outputs, and add provider-wide regression coverage. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Fix Crop validation issues. --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
### Description <!-- Describe your changes. --> Try to organize agent guidance so it is more extensible. E.g., add path-specific instructions, code review skill. Also moved skills to `.github/skills`. Added pointer to this location in AGENTS.md. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Goal is to improve agent code reviewing by providing guidance to follow. --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…1957) **[MLAS] NCHWc HardSwish fusion for MobileNetV3 FP32 CPU inference** **Implementation of the hardswish fusion kernel for mobilenetv3 models.** **Implementation changes done:** - Fused the decomposed HardSwish pattern (Mul(x, HardSigmoid(x))) into the preceding NCHWc Conv as a native HardSwish activation, eliminating two standalone nodes per occurrence. - Added TryFuseNchwcHardSwish in nchwc_transformer.cc to detect and apply the fusion when the HardSigmoid parameters match HardSwish values (α=1/6, β=1/2) and the Mul consumes the same conv output. **Performance:** **Performance numbers on AMD Ryzen AI 9 365 (Strix Point)** NOTE: Performance taken on 31st July <img width="927" height="176" alt="image" src="https://github.com/user-attachments/assets/6246ceb6-fb2d-4d95-8768-dc808d1ee126" /> **STRIX 365 configration:** AMD Ryzen AI 9 365 (Strix Point) w/ Radeon 880M **Target Model:** MobilenetV3-Large and MobilenetV3-Small (FP32, CPU) **Unit Tests** **test_nchwc_hardswish.cpp** - Verifies TryFuseNchwcHardSwish correctly fuses the Mul(x, HardSigmoid(x)) diamond pattern into the preceding NCHWc Conv - Confirms fusion is not applied when α/β do not match HardSwish values (standalone HardSigmoid falls through to standard activation path) - Top-1 accuracy unchanged after fusion
## [WebGPU] PagedAttention: direct paged decode, fused paged prefill, Unpack/Repack skip (Phase 2 partial) This PR is the Phase 2 follow-up to microsoft#31611. It replaces the "always gather + always Unpack/Repack" v1 fallback with two paged-aware FlashAttention programs that read the paged KV cache directly, and a fast path that lets FA consume the packed varlen Q buffer without materializing padded BSNH scratch. Net effect: **~2× faster decode, ~1.15× faster uniform prefill, ~1.3× faster varlen prefill** on the shape matrix below, with no regressions. The Phase 1 gather-then-flash fallback shipped in microsoft#31611 remains intact and still runs on adapters / configs where the paged-aware shaders can't safely dispatch (see `Correctness invariants` below). ### What's shipped 1. **Direct paged split-reduce decode.** `FlashAttentionPagedDecodeQKV` + `FlashAttentionPagedDecodeVxReduce` index `key_cache` / `value_cache` directly through `block_table`. Selected when `max_seqlen_q < 32` — mirrors the dense-FA split-reduce threshold. Eliminates the dense K/V scratch and its gather bandwidth for every decode step. 2. **Fused paged prefill.** `FlashAttentionPagedPrefillProgram` is a straight port of the dense-FA prefill shader's shared-memory path with page-table-aware K/V tile loads (`bert/flash_attention_paged_prefill.wgsl.template`). Supports fp16, BSNH Q, packed varlen Q (`q_varlen` template variant), and variable-Q-length causal masking via `seqlen_k` + `seqlens_q`. No attention_bias / head_sink / TurboQuant. 3. **Unpack/Repack skip fast paths.** When direct paged attention runs, we can hand FA a rank-4 view over the raw packed Q buffer instead of allocating padded BSNH scratch: - **Uniform mode** (`B * max_seqlen_q == token_count`): view is `[B, max_seqlen_q, N, H]`. Covers decode, `B==1` prefill, and equal-length batched prefill (the common continuous-batching case). - **Varlen mode**: view is `[token_count, 1, N, H]` plus `cumulative_seqlens_q`; only the fused paged-prefill shader can index it (`q_varlen`). Skipping Unpack+Repack removes 2 dispatches (~300–500 µs of CPU dispatch cost per Run on D3D12) plus a `B * max_seqlen_q * hidden * 2 B` scratch allocation (tens of MB at long prefill). ### Dispatch-count reduction | Route (no rotary, non-packed) | microsoft#31611 (merged) | This PR (shm-path adapters) | |---|---|---| | **Decode** (`max_seqlen_q < 32`) | Scatter + Gather + UnpackQ + DecodeQKV + DecodeVxReduce + Repack = **6** | Scatter + PagedDecodeQKV + PagedDecodeVxReduce = **3** | | **Prefill** (`max_seqlen_q ≥ 32`) | Scatter + Gather + UnpackQ + FlashAttention + Repack = **5** | Scatter + FlashAttentionPagedPrefill = **2** | Decode's FA is 2 kernels (split-K: QKV + VxReduce); prefill's FA is 1 kernel (`FlashAttentionProgram`). On configs where `ShouldRunFusedPagedPrefill` rejects (fp32, `head_size > 256`, `block_size < max_k_step`), the prefill row falls back to the 5-dispatch microsoft#31611 cascade; decode's direct paged split-reduce path has no such gate. Neither route is adapter-gated — the paged shaders use no subgroup intrinsics and run on every WebGPU adapter that meets the fp16 / shm-budget / alignment predicates. ### Correctness invariants **Prefill selection** consults one shared predicate: ```cpp bool ShouldRunFusedPagedPrefill(context, is_fp16, max_seqlen_q, head_size, block_size); ``` It rejects (→ gather-then-flash fallback) when any of: - `!is_fp16` — only fp16 variant is compiled today. - `max_seqlen_q < 32` — decode uses the split-reduce programs instead. - `head_size` exceeds the workgroup shared-memory budget (fp16: `head_size > 256`). - `block_size < max_k_step` — the fused shader assumes one K/V tile lives in one paged block (one `block_table` lookup per tile). `paged_attention_helper` only enforces `block_size >= 16` power-of-two; e.g. `block_size=16` with fp16 `head_size<=128` (`max_k_step=32`) would splice into a physically-adjacent block that isn't the next entry in the table. The fused paged-prefill shader uses only workgroup shared memory (no subgroup intrinsics), so there is no adapter-class gate — subgroup adapters (Qualcomm / AMD / Intel with subgroups) take the paged shm kernel directly instead of falling back to gather + dense-FA-subgroup. Because the same predicate gates the "skip `RunGatherKV`", "skip `q_padded` scratch", and "select fused shader" decisions, the three cannot drift. **Decode selection** (`max_seqlen_q < 32`) is a pure shape check — no adapter, dtype, or block-size gate. The direct paged split-reduce kernels (`FlashAttentionPagedDecodeQKV` + `FlashAttentionPagedDecodeVxReduce`) are the sole decode path when the kernel dispatches at all (fp16 is enforced at kernel registration, so no fp32 fallback is possible). Unlike fused prefill, the decode kernels do one `block_table` lookup per K/V slot rather than per tile, so they have no `block_size` alignment requirement. **WGSL correctness gotcha handled** in the fused prefill shader. `cumulative_seqlens_q` is `array<i32>` but row indices are `u32`. Both `loadq` and `writeo` explicitly cast (`u32(cumulative_seqlens_q[b]) + q_idx`); without the cast, tint surfaces the type-resolution failure as an opaque `absl::…raw_hash_map<>::at` at runtime. ### Performance Machine: dev-box discrete WebGPU adapter (D3D12), 24-core host. Google Benchmark harness at `onnxruntime/test/onnx/microbenchmark/paged_attention.cc`, `--benchmark_min_time=0.3s`, wall-clock timing via `UseManualTime()`. Earlier revisions of this PR included an `ORT_WEBGPU_PAGED_ATTENTION_USE_FUSED` env-var kill switch used for A/B measurement against the microsoft#31611 cascade. That toggle has been removed (the direct/fused paths are selected internally by shape and config; the numbers below are the reason). The A/B was performed by temporarily broadening the toggle locally to also force `use_direct_paged_decode=false` and `skip_unpack_repack=false`, so fused=0 exercised the exact gather-then-flash cascade shipped in microsoft#31611. All numbers below are with that broadened toggle; the broadening was reverted before final push. Column meanings: **nH** = num query heads, **nKV** = num KV heads, **H** = head dim. Shape families: - MHA_H64 (nH=16, nKV=16, H=64), MHA_H128 (nH=16, nKV=16, H=128) - GQA_Qwen (nH=14, nKV=2, H=128), GQA_Llama (nH=32, nKV=4, H=128) #### Decode (16 shapes) | Shape (B/nH/nKV/H/past) | this PR (µs) | microsoft#31611 (µs) | Speedup | |---|---:|---:|---:| | 1/16/16/128/2048 | 669 | 3612 | **5.40×** | | 2/16/16/128/512 | 627 | 3248 | **5.18×** | | 1/16/16/64/2048 | 861 | 3835 | **4.45×** | | 2/16/16/64/2048 | 907 | 3617 | **3.99×** | | 2/16/16/64/512 | 603 | 1369 | 2.27× | | 2/16/16/128/2048 | 2129 | 4816 | 2.26× | | 1/16/16/128/512 | 607 | 1150 | 1.89× | | 2/32/4/128/2048 | 1590 | 3010 | 1.89× | | 1/14/2/128/2048 | 979 | 1601 | 1.64× | | 2/32/4/128/512 | 656 | 990 | 1.51× | | 1/16/16/64/512 | 576 | 843 | 1.46× | | 2/14/2/128/512 | 694 | 984 | 1.42× | | 1/14/2/128/512 | 600 | 757 | 1.26× | | 2/14/2/128/2048 | 970 | 1194 | 1.23× | | 1/32/4/128/512 | 643 | 751 | 1.17× | | 1/32/4/128/2048 | 1434 | 1437 | 1.00× | Range **1.00×–5.40×**, geomean ~2.0×. Biggest wins on long-past MHA (H=128, past=2048) where gather bandwidth dominated. The one 1.00× row is a small-K/V-cache GQA case where gather cost was already low. #### Uniform prefill (24 shapes) Range **1.01×–1.25×**, geomean ~1.13×. Highlights (all wins): | Shape (B/nH/nKV/H/T) | this PR (µs) | microsoft#31611 (µs) | Speedup | |---|---:|---:|---:| | 1/32/4/128/128 | 1225 | 1530 | **1.25×** | | 2/14/2/128/128 | 1111 | 1385 | **1.25×** | | 2/16/16/64/128 | 766 | 948 | 1.24× | | 2/32/4/128/512 | 9723 | 12054 | 1.24× | | 1/16/16/128/128 | 856 | 1051 | 1.23× | | 2/16/16/128/1024| 17296 | 21029 | 1.22× | | 1/14/2/128/128 | 718 | 877 | 1.22× | | 2/32/4/128/128 | 1640 | 1966 | 1.20× | | 1/16/16/64/128 | 750 | 891 | 1.19× | | 2/16/16/128/128 | 1294 | 1491 | 1.15× | *(14 more rows 1.01×–1.15×; full log in tree.)* Short-T shapes gain most from Unpack/Repack skip; long-T shapes are dominated by FA compute time. #### Varlen prefill (12 shapes, halving q_lens = `{max_T, max_T/2, max_T/4, …}`) Range **1.14×–1.73×**, geomean ~1.29×. | Shape (B/nH/nKV/H/maxT) | q_lens | this PR (µs) | microsoft#31611 (µs) | Speedup | |---|---|---:|---:|---:| | 4/16/16/128/512 | `{512,256,128,64}` | 5987 | 10375 | **1.73×** | | 4/14/2/128/512 | `{512,256,128,64}` | 5312 | 7815 | **1.47×** | | 4/32/4/128/512 | `{512,256,128,64}` | 11165 | 15427 | **1.38×** | | 4/16/16/128/1024 | `{1024,512,256,128}` | 20127 | 27661 | **1.37×** | | 2/32/4/128/512 | `{512,256}` | 8345 | 10732 | 1.29× | | 2/16/16/128/1024 | `{1024,512}` | 15121 | 19084 | 1.26× | | 4/14/2/128/1024 | `{1024,512,256,128}` | 17685 | 21562 | 1.22× | | 4/32/4/128/1024 | `{1024,512,256,128}` | 39571 | 47468 | 1.20× | | 2/14/2/128/1024 | `{1024,512}` | 13141 | 15430 | 1.17× | | 2/16/16/128/512 | `{512,256}` | 4528 | 5235 | 1.16× | | 2/32/4/128/1024 | `{1024,512}` | 28871 | 33383 | 1.16× | | 2/14/2/128/512 | `{512,256}` | 4067 | 4645 | 1.14× | Wins grow with batch size — bigger B means more of the padded-BSNH round-trip gets eliminated (B=4/maxT=512 packs only 46.9% of `B·maxT` tokens; the padded scratch microsoft#31611 allocates is >2× bigger than the actual data). ### Tests - `onnxruntime/test/contrib_ops/paged_attention_op_test.cc` `PagedAttention.EndToEnd_*` — 12/12 non-CUDA tests pass. Covers MHA, GQA, single/multi-batch, variable past lengths, empty tokens, packed QKV, rotary, mixed prefill+decode, cache aliasing via IO-binding. New: `EndToEnd_Prefill_MultiBatch_Varlen_Fused` (B=2, token_count=48 with q_lens (32,16), head_size=128, MHA) — regression test for the fused varlen prefill path. - Micro-benchmark harness `onnxruntime/test/onnx/microbenchmark/paged_attention.cc` — 52 registered shapes (16 decode + 24 uniform prefill + 12 varlen prefill). ### Related - Phase 1 (v1 fallback): microsoft#31611 (merged) - Schema extensions: microsoft#29912 (merged) - Design doc: `docs/design/webgpu_paged_attention.md` (updated in this PR) --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com>
…ileClip-S0 model (microsoft#31958) **[MLAS] AVX-512 optimizations for MobileClip-S0 FP32 CPU inference** * Added a **16-wide AVX-512 Erf kernel** for standalone ONNX Erf operations, replacing the previous 8-wide implementation on AVX-512 hardware. * Implemented a **single-pass 16×16 AVX-512 NCHWc reorder transpose**, replacing the SSE2-based 4-wide sub-transpose approach for block-16 data reordering. The performance numbers taken in STRIX 365 with different thread configurations: <img width="974" height="116" alt="image" src="https://github.com/user-attachments/assets/a330b407-9d61-45f7-b6a5-9a08a774d6fc" /> NOTE: The performance numbers were tested on July 31st **STRIX 365 configuration:** AMD Ryzen AI 9 365 (Strix Point) w/ Radeon 880M **Target Model:** MobileClip-S0 (FP32, CPU) **Unit tests were added:** **test_erf.cpp** - MlasComputeErf vs std::erf within polynomial accuracy tolerance — sweeps buffer lengths straddling the 16-lane boundary to cover both the AVX-512 main loop and masked-tail path - Direct comparison of MlasErfKernelAvx512F vs the base MlasErfKernelFma3 with ≤1 ULP agreement — covers NaN propagation, ±inf, denormals, saturation - Measured divergence on AVX-512 hardware: 0 ULP (bit-exact); 1 ULP is kept as the cross-microarchitecture contract **test_reorder_input.cpp** - MlasReorderInputNchw (NCHW→NCHWc) vs scalar reference via memcmp - Sweeps channel counts 1–47: exact 16-channel blocks exercise the new MlasReorderInputNchwBlock16Avx512F fast path; partial blocks exercise the scalar tail - Multiple spatial sizes covered --------- Co-authored-by: Manogna-Sree <elisetti.manognasree@multicorewareinc.com>
### Description <!-- Describe your changes. --> ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. -->
## Summary This PR pins GitHub Actions to full-length commit SHAs for improved security and reproducibility and adds a 7 day cooldown to Dependabot configuration for GitHub Actions. This work is described in more detail at https://aka.ms/action-pinning. ## Why? Pinning actions to commit SHAs prevents supply-chain attacks where a tag could be moved to point to malicious code. This is a recommended security best practice per the [GitHub Actions security hardening guide](https://docs.github.com/en/actions/security-for-github-actions/security-guides/security-hardening-for-github-actions#using-third-party-actions). This change mitigates the risk of tag retargeting to malicious code as seen in incidents like the [tj-actions/changed-files action compromise](https://www.stepsecurity.io/blog/harden-runner-detection-tj-actions-changed-files-action-is-compromised) or [codfish/semantic-release-action compromise](https://www.stepsecurity.io/blog/supply-chain-compromise-codfish-semantic-release-action) and improves the integrity and reproducibility of the CI/CD pipeline. ## What changed? **Action pinning:** Third-party action references in `.github/workflows/` that used mutable tag-based references (e.g., `actions/checkout@v4`) have been updated to full-length commit SHAs with a version comment (e.g., `actions/checkout@<sha> # v4`) using the [pinact](https://github.com/suzuki-shunsuke/pinact) tool. References that were already pinned to a SHA, or that used immutable release tags, were left unchanged. **Dependabot configuration:** `.github/dependabot.yml` has been updated to ensure a `github-actions` package-ecosystem section is present with a `cooldown` configuration (`default-days: 7`). If the file did not exist, it was created. If a `github-actions` section already existed, only the `cooldown` block was added or its `default-days` value was increased to 7 if it was lower. The 7-day cooldown provides a window for the community to detect and report compromised releases before they are automatically proposed as updates, reducing exposure to supply-chain attacks via newly published malicious versions. ## Is this safe to merge? Yes. The pinned SHAs correspond to the same commits that the existing tags pointed to. No behavioral changes in action execution are introduced. You can verify the pinned SHA value using the GitHub REST API (e.g., the commit hash for `actions/checkout@v7` can be found in the `sha` property in the JSON response for `GET https://api.github.com/repos/actions/checkout/commits/v7`). ## Additional Information For more information, please see https://aka.ms/action-pinning
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
14 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated daily backmerge from ORT main to ovep-develop. No conflicts detected. Do NOT squash or rebase - use merge commit only.