GH-4012: share one settle budget across the stacked ack retry blocks

[jeremydmiller]

Addresses items 1 and 2 of #4012. Items 3-5 (per-transport terminal classification, broker-side redelivery-count bounding, JasperFx RetryBlock.ShouldRetry) stay open on that issue.

The multiplication bug

RetryBlock.MaximumAttempts bounds retries within one PostAsync. The durable completion path stacks two of them. Traced end to end rather than taken from the analysis doc:

DurableReceiver._completeBlock                              (RetryBlock, max 3)
  -> RabbitMqListener.CompleteAsync(Envelope)          :222
  -> RabbitMqChannelCallback.CompleteAsync             :63
  -> RabbitMqChannelCallback.Complete                        (RetryBlock, max 3)
  -> RabbitMqEnvelope.CompleteAsync                    :37
  -> RabbitMqListener.CompleteAsync(RabbitMqEnvelope)  :512  -> BasicAckAsync

Three by three is nine broker round trips for a single delivery, and neither block can see the other's count.

The fix

Envelope.AckAttempts rides the envelope so the layers share a budget — DurabilitySettings.MaximumAckAttempts, default 3.

Where the increment lives matters. It belongs to the innermost layer that actually issues the broker call, because that is the only layer that knows a round trip really happened. Outer layers only check. That has a deliberate consequence worth a reviewer's eye:

A transport that settles directly in Listener.CompleteAsync — no inner retry block — never increments, so the outer guard never trips and its behavior is byte-for-byte what it was. RabbitMQ gets the fix; every other transport is untouched until item 3 extends it per transport.

Exhausting the budget swallows rather than throws. An unsettled delivery is the recoverable outcome — the broker redelivers and the durable inbox deduplicates — and it is the same recovery the existing unknown-delivery-tag branch already relies on. Throwing would just feed the enclosing RetryBlock.

Honest scope: this cannot bound a redeliver → dedupe → re-ack loop. Every redelivery constructs a brand new envelope, so the counter starts over. Only a broker-side delivery count (redelivered/x-death, ApproximateReceiveCount, DeliveryCount) can do that — item 4, and it is called out in the xml-docs so the next reader doesn't over-trust the counter.

The pooling guard blind spot

envelope_pool_tests.Reset_zeroes_every_settable_property reflected with default binding — public only — which left every internal { get; set; } property on Envelope outside the guard. Widened to NonPublic, and the internal properties are now stamped in the arrange block so the assertions aren't vacuous.

That blind spot was not theoretical. It immediately caught BatchPendingSettled missing from Envelope.Reset() since CritterWatch#942.

Reachability, checked before claiming anything: Batch is only ever assigned in the new Envelope(IEnumerable<Envelope>) constructor, never on a pooled envelope, and SettleBatch early-returns when Batch == null. So a stale flag cannot bite today — this is a latent gap, not a live bug. Fixed anyway, because it is exactly the drift the guard exists to catch and the next batching change could make it reachable.

Tests

• shared_ack_attempt_budget — a listener that spends the whole budget in one call (the Rabbit shape) is called once, not three times; a non-participating transport keeps its existing retry behavior; the budget is configurable.
• envelope_ack_attempt_budget — the counter refuses past the maximum and does not keep incrementing after refusal.
• The widened pooling guard covers AckAttempts and BatchPendingSettled.

Red-checks performed on both halves:

• Setting maximumAckAttempts = int.MaxValue (guard disabled) fails 2 of 3 budget tests.
• Removing the BatchPendingSettled = false; line produces Envelope.BatchPendingSettled not reset: expected False, got True.

Verification

• dotnet build wolverine.slnx -c Release -f net9.0 — clean
• Full CoreTests: 2487 passed, 0 failed, 2 skipped
• Full Wolverine.RabbitMQ.Tests against docker-compose Rabbit: 506 passed, 0 failed (11m14s). Worth noting both of the failures I'd expected from prior runs — multi_tenancy_through_virtual_hosts and end_to_end.use_direct_exchange_with_binding_key — passed here too.

Not covered

No Rabbit integration test of the budget: forcing repeated real ack failures needs broker-side channel manipulation beyond what the existing harness offers. The arithmetic is pinned at the DurableReceiver seam instead, which is where the multiplication actually happened.