fix(om): honor background posts requested while an acknowledgement is pending #290

Problem

OutboxProcessor<TOutbox>.PostInBackgroundAsync can be lost when it arrives while a background dispatch run has delivered items and is waiting for its acknowledgement. The newly committed item is not dispatched promptly. It waits for the dispatch timer's next period, which is OutboxProcessorOptions.RetryDelay (default 2 minutes), or for a consumer-owned fallback.

The flow at egil-orleans-messaging/v1.1.3 (Egil.Orleans.Messaging/src/Egil.Orleans.Messaging/Outboxes/OutboxProcessor.cs):

commit A → PostInBackgroundAsync → EnsureDispatchTimer(0) → DispatchInBackgroundAsync delivers A
                                                            pendingAcknowledgement = batch(A)
commit B → PostInBackgroundAsync → EnsureDispatchTimer(0)   (B is now in the outbox)
           a dispatch tick finds pendingAcknowledgement != null
           → "Defer instead of posting the same durable snapshot again" (:229-237), returns
ack A    → AcknowledgeInBackgroundAsync → ReconcileRetryStateAsync
           B still pending → EnsureDispatchTimer(options.RetryDelay) (:428)
B        → dispatched only after RetryDelay

PostAsync already handles the equivalent case for a foreground post during an active drain, through drainRequested (:98-103) and ScheduleRequestedDrain (:435). PostInBackgroundAsync has no equivalent, so a request that arrives during the delivery-to-acknowledgement window is absorbed by the defer branch. The acknowledgement then re-arms at retry cadence, as if the remaining item had failed.

Consumer impact

For The League's CompetitionEditionFactsGrain commits each accepted live score revision and calls PostInBackgroundAsync. When a score changes twice in quick succession while the downstream workflow is busy, the second revision lands in this window. It reaches the Live Projection only when the grain's own 1-minute durable wake-up fires. Without that wake-up, it would wait the full 2-minute RetryDelay.

Probe evidence from a deterministic For The League reproduction (message 3 is the second revision):

16:17:58.789 commit-post pending=[2]
16:17:58.790 deliver 2
16:17:58.791 commit-post pending=[2,3]
16:17:58.793 acknowledged [2] still-pending=[3]
             (no "deliver 3" within the 5 s bound; arrives after the 1 min fallback)

With RetryDelay = 1 s, message 3 is delivered 1 ms after the acknowledgement. That confirms the follow-up depends on the timer period, not on a prompt request. For The League is setting a short RetryDelay as a stopgap until this is fixed.

Work

• Make a PostInBackgroundAsync (and a ReceiveReminderAsync tick) that arrives while a dispatch run or a pending acknowledgement owns the outbox record the request instead of losing it. For example, set drainRequested in the defer branch, and in the background post path when an acknowledgement is pending.
• After AcknowledgeInBackgroundAsync completes, honor a recorded request with an immediate dispatch (ScheduleRequestedDrain), so ReconcileRetryStateAsync's RetryDelay re-arm applies only when nothing new was requested.
• Keep the existing guarantees:
  • one drain at a time;
  • no re-post of a snapshot whose acknowledgement is still pending;
  • acknowledgement as a separate turn, so the non-reentrant deadlock the comment at :244-248 describes does not come back;
  • RetryDelay cadence for genuinely failed or unacknowledged items.

Acceptance

• A test posts in the background, holds the first delivery's acknowledgement, commits a second item and posts again in the background, then releases the acknowledgement. The second item is delivered promptly, not after RetryDelay. The test runs with a RetryDelay long enough (minutes) that only a prompt path can pass, and fails on v1.1.3.
• A failed delivery still retries after RetryDelay, not immediately in a loop.
• The snapshot whose acknowledgement is pending is never posted twice.
• Existing outbox processor tests pass, including the foreground PostAsync-during-drain behavior.
• Released as an egil-orleans-messaging patch version so For The League can drop its RetryDelay stopgap.

Reviewed against egil-orleans-messaging/v1.1.3 (0280d05b040dd9a599e06d6f93da73a280e33637). No later commit on main (8d4de16752434a94f0294245ef55ba2372efec1b) touches Outboxes/.