Fix silent message loss when envelope mapper throws (RabbitMQ + MQTT)#2511
Merged
jeremydmiller merged 1 commit intoApr 14, 2026
Merged
Conversation
…JasperFx#2510) When the incoming envelope mapper throws, both WorkerQueueMessageConsumer (RabbitMQ) and MqttListener (MQTT) previously logged the error and acked the broker delivery, silently dropping the message. No broker DLQ, no entry in wolverine_dead_letters, no redelivery — the message was simply lost. This surfaces in real deployments when a queue is shared with a foreign producer (e.g. MassTransit) whose headers the mapper cannot read, when a custom mapper throws, or when header decoding hits an unexpected AMQP type. Both listeners now attempt ISupportDeadLetterQueue.MoveToErrorsAsync on their IReceiver, synthesizing an envelope Id first so the dead-letter store has a primary key. RabbitMQ falls back to BasicNack(requeue:false) so the broker's DLX captures the message when the receiver path isn't available. MQTT always acks after attempting the dead-letter move to avoid a poison-redelivery loop (MQTT has no broker DLQ). Both sites contain every outbound call in try/catch so failures inside the fix path cannot propagate out of the consumer callback. MqttEnvelope now sets Destination in its constructor, mirroring how Data and MessageType are populated. Previously Destination was set by DurableReceiver during MarkReceived, which is bypassed in the mapper- exception path and caused MessageDatabase.MoveToDeadLetterStorageAsync to NullReferenceException when formatting the destination URI. Integration tests cover both transports: the RabbitMQ test publishes a raw AMQP message to a queue with a throwing mapper and asserts the broker DLQ receives it; the MQTT test uses a SQL Server durable inbox and asserts an entry appears in wolverine_dead_letters with the original exception type. Closes JasperFx#2510
Member
|
@BlackChepo You've been on fire! But also wish you didn't have quite so much to go fix lately:/ I'll deal w/ the CI problems |
Contributor
Author
|
Thanks Jeremy! The good news is we're finding them before users do. 😄 Happy to keep contributing fixes as I run into things |
3 tasks
jeremydmiller
added a commit
that referenced
this pull request
Apr 14, 2026
Fix CI failures from PR #2511
This was referenced Apr 15, 2026
This was referenced Apr 25, 2026
This was referenced May 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2510
Summary
When the incoming envelope mapper threw, both RabbitMQ's
WorkerQueueMessageConsumerand MQTT'sMqttListeneracked the broker delivery and dropped the message. No brokerDLQ, no entry in
wolverine_dead_letters, no redelivery — the message was simply lost. Triggered in the wild by foreign producers (e.g. MassTransit) whose headers themapper can't decode, custom mappers that throw, or header decoding on hosts with a non-UTF8 system codepage.
Both listeners now:
envelope.Id(white lie — the mapper failed before populating it, butMoveToErrorsAsynckeys by Id).ISupportDeadLetterQueue.MoveToErrorsAsyncon the receiver so the message lands inwolverine_dead_letterswhen the durable inbox is configured.BasicNack(requeue: false)so the broker's DLX captures the message even without durable inbox.MqttEnvelopenow setsDestination = topic.Uriin its constructor, matching howDataandMessageTypeare already populated. PreviouslyDestinationwas set byDurableReceiver.MarkReceived, which is bypassed when the mapper throws — causingMessageDatabase.MoveToDeadLetterStorageAsyncto NRE onenvelope.Destination!.ToString().Transport behavior comparison (before → after)
BasicAck→ silent lossMoveToErrorsAsync→ broker DLX + DB when configured_complete.PostAsync→ silent lossMoveToErrorsAsync→ DB (when durable inbox) + PUBACKTest plan
Bug_mapper_exception_routes_to_dlqunderWolverine.RabbitMQ.Tests.Bugs— publishes a raw AMQP message to a queue with a throwingIRabbitMqEnvelopeMapperand asserts the broker DLQ receives it.Bug_mapper_exception_routes_to_dlqunderWolverine.MQTT.Tests.Bugs— durable inbox on SQL Server + throwingIMqttEnvelopeMapper; asserts anentry appears in
wolverine_dead_letterswithExceptionTypecontainingInvalidOperationException.mainwith a "(silent loss)" exception and pass on this branch.Files changed
src/Transports/RabbitMQ/Wolverine.RabbitMQ/Internal/WorkerQueueMessageConsumer.cssrc/Transports/MQTT/Wolverine.MQTT/Internals/MqttListener.cssrc/Transports/MQTT/Wolverine.MQTT/MqttEnvelope.cs(1 line — setDestinationin constructor)src/Transports/MQTT/Wolverine.MQTT.Tests/Wolverine.MQTT.Tests.csproj(addedWolverine.SqlServerproject reference)src/Transports/RabbitMQ/Wolverine.RabbitMQ.Tests/Bugs/Bug_mapper_exception_routes_to_dlq.cssrc/Transports/MQTT/Wolverine.MQTT.Tests/Bugs/Bug_mapper_exception_routes_to_dlq.cs