Harden ordered-view bulk index builds: memory budget, cancellable sort, bounded tail replay - #679
Merged
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR hardens the ordered-view bulk index rebuild path under extreme log sizes by adding memory-bounded bulk-build selection, making the internal sorts cooperatively cancellable, and bounding/rescheduling tail replay to ensure forward progress without long owner-thread stalls. It also adds targeted unit tests to validate parity and the new abandonment/forced-replay behaviors.
Changes:
- Adds a bulk-build peak-memory estimator and a default memory budget that triggers a safe fallback to the bounded-memory delegating build path (and always delegates for Keyword joins).
- Replaces
Array.Sortin ordered-view sorting paths with a cancellation-aware introsort (ColumnDirectSort.CancellableSort) plus a shared cancellation check stride (CooperativeCancellation). - Extends rebuild adoption to support bounded tail replay with abandonment + a breach counter to force replay after repeated abandons, and adds deterministic concurrency tests to cover the behavior.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| tests/Unit/EventLogExpert.Runtime.Tests/LogTable/OrderedView/OrderedViewTailReplayTests.cs | Adds unit tests for tail-replay abandon/forced-replay/hold-release scenarios. |
| tests/Unit/EventLogExpert.Runtime.Tests/LogTable/OrderedView/OrderedViewBulkBuildTests.cs | Adds a memory-budget fallback parity test to ensure identical ordering vs bulk build. |
| tests/Unit/EventLogExpert.Runtime.Tests/LogTable/CancellableSortTests.cs | Adds cancellation + parity tests for the new cancellable sort implementation. |
| src/EventLogExpert.Runtime/LogTable/OrderedView/OrderedViewWriter.cs | Introduces tail replay budget/breach limit wiring and rescheduling behavior on abandoned tails. |
| src/EventLogExpert.Runtime/LogTable/OrderedView/OrderedViewState.cs | Implements memory-budgeted bulk build selection and tail measurement/abandon behavior for adoption. |
| src/EventLogExpert.Runtime/LogTable/OrderedView/ChunkedOrderIndex.cs | Switches cancellation polling to the shared cooperative cancellation stride/mask. |
| src/EventLogExpert.Runtime/LogTable/ColumnDirectSort.cs | Adds cancellable introsort and routes existing sort sites through it. |
| src/EventLogExpert.Runtime/Concurrency/CooperativeCancellation.cs | Centralizes the cancellation check stride/mask constants used across hot loops. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Joseph Schick (jschick04)
marked this pull request as ready for review
August 13, 2026 19:44
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hardens the ordered-view bulk index build path for the large-log extreme, in three independent, order/rank-preserving changes (no public API change):
Memory-bounded bulk build -
OrderedViewState.BuildIndexestimates peak transient bytes and falls back to the bounded-memory delegating insert path when a build would exceed available heap headroom (and always for unbounded Keyword joins), so a large rebuild can't OOM.Cancellable sort - replaces
Array.SortinColumnDirectSort(both the permutation sort and the dense-rank string sort) with a token-checking introsort that throws on cancellation (never returns a partial array), so a superseded build stops promptly instead of running an uncancellable multi-second sort. Consolidates the repeated cancellation-check stride intoConcurrency.CooperativeCancellation.Bounded tail replay + forward progress -
OrderedViewState.TryAdoptRebuildabandons an over-budget owner-thread tail replay and reschedules; a breach counter forces a bounded owner-thread replay after repeated abandons, preventing both owner-thread stalls and a sustained-append livelock.Tests - +3 files (~26 tests): identical-permutation parity across 756 sort configs, mid-sort cancellation at both sort sites, the memory-budget fallback, and the tail-replay abandon / forced-fallback / hold-clear paths (deterministic concurrency tests; key ones verified to fail under their target regression). Before/after benchmark is perf-neutral with identical allocation.