Restore send ordering in BatchedSender, and order tracked records honestly (GH-3825)

[jeremydmiller]

Closes #3825.

The issue said "session handling, 2 of 6 fail deterministically." The symptom half was right; the cause was not. Nothing was ever lost — every message arrived, in the wrong order. And it was wrong for two independent reasons.

Recording the handler's own execution order next to session.Received split them apart:

run 0 BUFFERED tracked : B1, B6, B5, B3, B4, B2
run 0 BUFFERED handled: B1, B2, B3, B4, B5, B6   <-- actual processing was fine

1. BatchedSender lost enqueue order

The serializing stage ran at Environment.ProcessorCount, so envelopes reached the batching block in serialization-completion order rather than enqueue order.

git log -L on the constructor shows this was not a design decision. Before the Channels rewrite (8576ce526, "First cut at switching over to Channels in place of TPL DataFlow") it was an ActionBlock:

_serializing = new ActionBlock<Envelope>(async e => { ... },
    new ExecutionDataflowBlockOptions { CancellationToken = _cancellation, ... });

TPL Dataflow's MaxDegreeOfParallelism defaults to 1, so that block was strictly ordered. The port raised it to Environment.ProcessorCount and the guarantee went with it — silently breaking the FIFO contract behind Azure Service Bus sessions, SQS FIFO message groups, and global partitioning, for every transport that sends through BatchedSender.

Demonstrated with a buffered publisher and an inline one side by side, same group id:

	run 0	run 1	run 2
buffered	B1, B6, B5, B4, B2, B3	B1, B4, B3, B2, B5, B6	B5, B3, B4, B6, B2, B1
inline	I1..I6	I1..I6	I1..I6

Pinned back to 1. Everything downstream was already serial by default (Endpoint.MessageBatchMaxDegreeOfParallelism is 1).

2. TrackedSession mis-reported the order of its own records

.OrderBy(x => x.SessionTime)   // whole milliseconds

SessionTime is _stopwatch.ElapsedMilliseconds. An entire receive batch shares one value, and the stable sort then falls back to enumerating _envelopes — a Guid-keyed cache with no relationship to when anything happened. Every ordering assertion written against Received/Sent anywhere in the suite was riding on that.

EnvelopeRecord now carries a monotonic Sequence assigned in the constructor (so every construction site is covered), and both AllRecordsInOrder overloads sort on it.

Verification

• New BatchedSenderTests.preserves_enqueue_order_through_to_the_outgoing_batch uses a serializer with descending per-envelope delays, so any parallelism in that stage reorders every run rather than occasionally. Confirmed red before the change, green after.
• end_to_end 6/6, untagged from Category=Flaky.
• CoreTests 2247/0 · ASB 315/0 · MartenTests 548/0 · wolverine.slnx -c Release clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WHAuhdWS3XeAk16swV9G8m