Repository navigation
feat(messaging): deliver outbox messages - #11285
Open
ReubenBond wants to merge 135 commits into
Open
ReubenBond wants to merge 135 commits into
ReubenBond wants to merge 135 commits into
Conversation
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
In-flight inbox and outbox pump turns do not revalidate callback generation and physical ownership after recovery, allowing stale callbacks to mutate current state.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 2
Open (2)
What changed in this PR
Adds the durable outbox layer above the journaled inbox, including durable ownership, delivery pumping, retries, dead letters, and extensive contract/functional tests.
Changes:
- Adds journaled outbox scheduling and delivery coordination.
- Extends journaling with observer and participant lifecycle APIs.
- Adds durable messaging routing, serialization, diagnostics, and test infrastructure.
| File | Description |
|---|---|
src/Orleans.DurableMessaging/* |
Durable messaging runtime, contracts, routing, ownership, and delivery. |
src/Orleans.DurableMessaging/README.md |
Documents inbox/outbox behavior and layering. |
src/Orleans.DurableMessaging/Configuration/DurableInboxOptions.cs |
Adds messaging configuration and validation. |
src/Orleans.Journaling/IJournaledStateObserver.cs |
Adds journal boundary observer contract. |
src/Orleans.Journaling/IJournaledStateManager.cs |
Adds observer registration. |
src/Orleans.Journaling/IJournaledGrainParticipant.cs |
Adds feature participant contract. |
src/Orleans.Journaling/DurableGrain.cs |
Initializes journaled participants. |
src/api/Orleans.Journaling/Orleans.Journaling.cs |
Updates generated journaling API surface. |
test/Orleans.DurableMessaging.Tests/* |
Adds durable messaging contract, component, and functional tests. |
Orleans.slnx |
Includes durable messaging source and test projects. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Contributor
Code coverage
Report-only conclusion: current-main baseline stale. The newest successful coverage run tested ebfde71, not current main ceeebcf. Coverage combines every CI test matrix job, including providers, CodeGen, .NET 8/10, Linux, Windows, and macOS, using canonical physical source and branch identities. The comparison remains report-only while normal line and branch variance is calibrated. Coverage details |
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
A recovery race can commit outbox work without scheduling a durable owner.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 2
Open (2)
Resolved since last review (1)
ReubenBond
force-pushed
the
rb-special-system
branch
from
September 17, 2026 01:41
55fd4af to
d23dabb
Compare
ReubenBond
force-pushed
the
rb-special-system
branch
from
September 17, 2026 02:24
d23dabb to
8b4342c
Compare
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Inbox deletion does not block active interleaved pump or gate operations, allowing post-delete work or stale-generation failures.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 1
Open (1)
Resolved since last review (1)
ReubenBond
force-pushed
the
rb-special-system
branch
from
September 17, 2026 03:35
8b4342c to
e430bf3
Compare
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Remote delivery cancellation disposes its token source while in-flight attempts may still use it, creating a concurrency failure during ownership transitions.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 1
Open (1)
Resolved since last review (1)
ReubenBond
force-pushed
the
rb-special-system
branch
from
September 17, 2026 07:45
a30bdee to
4470a33
Compare
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Inbox owner clearing can race interleaved acceptance preparation and fence the activation instead of persisting the incoming message.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 1
Open (1)
Resolved since last review (1)
ReubenBond
force-pushed
the
rb-special-system
branch
from
September 17, 2026 08:25
4470a33 to
e9f7764
Compare
…port them Messaging fails the activation when a failed handler attempt staged changes, so they can't commit with its retry record, and decides that by comparing StagedChangeCount before and after the attempt. A JournaledPersistentState is mutated in place and serialized whole when a write captures it, so its changes never advance the count: a handler which mutated it and threw would have its partial state committed by the retry write. States which report every change now implement the internal IReportsStagedChanges: the built-in collections, values, and task completion sources, and the messaging inbox. While any other state is registered, StagedChangeCount is negative, which messaging already treats as staged, so a failed attempt recovers from the journal instead.
CompactProcessed removes processed records while enumerating them, which Dictionary supports on .NET Core 3.0 and later, and so on both frameworks Messaging targets. The test places expired records first and between retained ones, so compaction keeps enumerating after each removal.
Options validation accepts any positive DeduplicationWindow, including TimeSpan.MaxValue. The floor subtracts the window from the clock's ticks, which stays above long.MinValue for every representable time, and clamps at the minimum time. The test runs delivery, deduplication, and snapshot compaction with that window.
A handler attempt which fails after staging changes fails the activation, so recovery discards those changes, but it also discards the attempt's failure record. The message then reran with the same attempt count, so backoff and InboxRetryTimeout never applied: a handler which always failed after staging looped indefinitely, and so did any failure on a grain whose state can't report its changes. A graceful stop now journals a clean-stop record when messages are pending, and recovery charges one attempt to each due message unless the journal ends with that record. The charge commits before the pump runs a handler, and a recovery which finds a clean stop journals that it resumed, so a later unclean stop is charged. The happy path writes nothing extra; the cost falls on deactivation with pending work and on recovery. A crash also charges due messages which hadn't been dispatched, so attempts can overcount by one per unclean stop. A charge starts a message's retry window but dead-letters only a message whose window an earlier failure started, so one crash never dead-letters a message by itself.
The future bound, its exemption for recorded and floor-expired identities, and equality admission shipped without tests. Pin each branch, including receiver clock rollback, on a Durable Jobs clock which stands still between reading the bound and delivering.
The outbox stages each send in the grain's own journal write and delivers it once that write is durable, retrying with backoff on the Durable Jobs clock until the receiver answers Accepted, Duplicate or DeadLettered. A lost answer is redelivered and the receiver's deduplication answers it, so a rejection never becomes Delivered. Messages a grain sends to itself are staged in its own inbox in the same write. Sources can keep a message's payload as its only journaled copy and are told how it retired, and diagnostic events report inbox and outbox transitions. This layer implements the contracts which L1 defines, so it changes none of them: - Every attempt starts within OutboxRetryTimeout of the send, or of the latest attempt the receiver answered Backpressured, which proves it had not accepted the message. The expiration is journaled with the send, so an entry whose removal write failed after an acceptance cannot be redelivered after the receiver's DeduplicationWindow; it is dead-lettered instead. - Up to OutboxBatchSize attempts are in flight, each applied as it completes, so a hung receiver holds one slot instead of stalling every outbound message. An entry stays in flight until its outcome is applied, so it never has two attempts. - Withdrawal is explicit through IOutbox.Withdraw, and an attempt in flight for a withdrawn entry is ignored. A source which stops providing a message it owns dead-letters it, and a recovered reference which no source provides is dead-lettered loudly instead of being deleted as if withdrawn. - A status the sender can't interpret, such as one a newer receiver adds, leaves the outcome unknown and retries. - The Durable Jobs clock is the single retry authority: attempts call the receiver through an internal transport, which tests decorate to inject faults, instead of a silo-wide Polly pipeline on a second clock. - Attempts start before any write is awaited, so nothing interleaves between materializing a message and calling its receiver.
Every fault test scripted one fault for one message, so no test combined faults, crashes and time against the end-to-end contract. Generate schedules of sends, delivery faults, sender and receiver crashes, and clock advances, and check that no message's effect is ever applied twice and that, once faults stop, every message is processed exactly once in effect or dead-lettered at the sender after its retry timeout.
Deactivation cancels in-flight outbox attempts instead of waiting for a blocked receiver, so the sender can be gone before the test observed it. The durable wake-up and the recovered activation's delivery establish that the journal carried the message.
…line The receiver rejects a message whose redeliveries would outlast its deduplication record, so each outbox attempt carries the latest time the outbox may deliver it again: the expiration the attempt started under, which a backpressured answer restarts, plus the grain call response timeout, because Orleans drops a call older than that instead of invoking it. The receiver keeps nothing for a DeduplicationWindowExceeded rejection, so the outbox keeps the message as a dead letter, which can be sent again under its ID once the receiver's window covers the sender's retries, and retires it as Rejected with that code. The generated fault simulation adds a sender with a longer horizon and checks that a rejected message is never processed.
The outbox journals every send when it is staged, so it implements IReportsStagedChanges. Without it, every grain with an outbox would report unknown staged changes, and any failed handler attempt would fail the activation rather than retry.
…end's wake-up A retry write records NextAttemptAt without scheduling a job, because the wake-up which the send ensured before it became durable keeps rescheduling while the outbox holds the entry. The test pins that its run reschedules to the retry time and that recovery after a crash delivers the message once.
MessagingEvents.AllEvents is process-wide, so a probe which never disposes its subscription stays rooted after its cluster and receives later tests' events. The probe now keeps the subscription, and the bootstrap fixture disposes it with the cluster.
…stop A failed clean-stop write leaves the journal as a crash would. Recovery charges the interrupted attempt and a queued message which hadn't run one attempt each, dead-letters neither under a zero InboxRetryTimeout because neither had failed before, and processes both once afterward.
| private DateTimeOffset GetAdmissionFloor(DateTimeOffset now) | ||
| { | ||
| var nowTicks = now.UtcTicks; | ||
| var candidate = new DateTimeOffset(Math.Max(0, nowTicks - Options.DeduplicationWindow.Ticks), TimeSpan.Zero); |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Durable messaging stack: #11282 (contracts) → #11284 (inbox) → #11285 (outbox) → #10693 (integration, documentation, samples, NativeAOT smoke, benchmarks, and playground). This layer builds on #11284.
Problem
Grains need to commit business changes and outgoing intent together, recover delivery across crashes, and preserve executable application action when delivery stops with rejection or unknown outcome.
Solution
SentAtonce alongside business changes. Delivery starts after acknowledgement; attempts and recovery reuse the time. Retries end atSentAt + OutboxRetryTimeout, with the budget shorter thanDeduplicationWindow.IOutboxMessageSource.OnRetiredstages application terminal state and follow-up work atomically with removal. A self-addressed follow-up resumes through the inbox's durable wake-up after recovery. Callback failure preserves committed pending intent for recovery.Rationale
Journaling supplies atomic intent and retirement. Existing source callbacks and inbox wake-ups supply recoverable application action. Bounded new times and a durable floor make tracking reclaimable while preserving recorded outcomes across clock movement.