GH-4509: sweep expired handled envelopes on CosmosDb, and make DeleteAllHandledAsync work

[jeremydmiller]

Closes #4509. Thanks to @r0ss88 for the report — the diagnosis in the issue was exactly right, including the pointer at Bug_4286_dead_letter_expiration as the place for the test.

What was broken

MarkIncomingEnvelopeAsHandledAsync stamped a keepUntil that nothing ever read back:

• CosmosDbDurabilityAgent's only sweep was tryDeleteExpiredDeadLetters
• there is no equivalent of the RDBMS DeleteExpiredHandledEnvelopesCommand
• no ttl is written, and MigrateAsync creates the container with default ContainerProperties, so container TTL is off

So handled inbox documents were kept forever, bodies included, since mark-handled is a read/modify/replace that leaves body alone. And there was no way to clean up afterwards either: DeleteAllHandledAsync threw NotSupportedException, so the built-in clear-handled command failed outright on this provider.

It bites harder on Cosmos than on a SQL store, as the issue notes: IncomingMessage.PartitionKey is the envelope's destination, so every handled envelope for a listening endpoint piles into one logical partition — 20 GB hard ceiling, 10k RU/s cap. FetchCountsAsync, which CheckHealthAsync calls on every health check, counts that pile cross-partition too.

What this adds

• tryDeleteExpiredHandledEnvelopes, mirroring the dead-letter sweep, on its own HandledMessageCleanupPollingTime cadence and bounded by HandledMessageCleanupBatchSize / HandledMessageCleanupMaxBatchesPerCycle. Those three knobs read as general durability settings but were referenced only from Wolverine.RDBMS.DurabilityAgent — on Cosmos they did nothing at all, which the issue also flagged.
• DeleteAllHandledAsync implemented with the same query minus the keepUntil predicate.

Unlike the dead-letter sweep this query is necessarily cross-partition — dead letters share one partition key, handled inbox documents do not — so each delete carries the document's own partition key, which is why the projection selects it.

The comparison has to be on timestamps, not strings

This is the part worth reviewing. keepUntil is persisted as an ISO-8601 string that keeps whatever offset it was written with; documents in the test container carry -05:00. Cosmos compares two strings lexicographically, so the obvious form:

AND c.keepUntil < @now          -- @now rendered as UTC

is meaningless the moment the offsets differ. The first version of this patch did exactly that, and it deleted a handled document half an hour short of its expiry — caught only because the test seeds a deliberately-retained document alongside the expired one. DateTimeToTimestamp normalises both sides to epoch milliseconds and reads the offset, so it is also correct for documents already written by an older version.

A latent instance of the same bug, deliberately not touched here: tryDeleteExpiredDeadLetters compares c.expirationTime the same string-wise way. Bug_4286_dead_letter_expiration passes only because its keeper is a full day out, so the dates differ and the lexicographic comparison happens to land correctly. Worth its own issue.

On the tests

Assertions are by envelope id, not through FetchCountsAsync().Handled. A running Wolverine host handles its own control traffic, so it mints fresh handled documents with the default five-minute retention while the test watches — the count never reaches zero and says nothing about whether the seeded document was swept. That is #4499's lesson again: a count does not tell you whether the rows are the same rows. The first draft asserted the count and failed for that reason rather than the real one.

Three documents per sweep test — one expired handled, one handled but well inside its window, one never handled — and only the first may disappear. Both tests fail on unmodified main.

Verification

CosmosDbTests 125 green against the emulator (Testcontainers self-provisions it). Clean pinned build: dotnet build wolverine.slnx -c Release -f net9.0.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VDUrBeB4tTnKj4AExCS1nj