Skip to content

[Mocha] Fix Postgres outbox dropping fault replies to a caller's reply queue - #10529

Merged
tobias-tengler merged 6 commits into
mainfrom
tte/mocha-outbox-fault-reply
Oct 14, 2026
Merged

tobias-tengler merged 6 commits into
mainfrom
tte/mocha-outbox-fault-reply

Conversation

@tobias-tengler

@tobias-tengler tobias-tengler commented Oct 12, 2026 •

Copy link
Copy Markdown
Member

When a request handler fails, Mocha answers the caller with a NotAcknowledgedEvent of kind MessageKind.Fault. With the Postgres outbox, the outbox processor only routed MessageKind.Reply through the transport's reply dispatch endpoint. A fault took the generic dispatch lookup instead, which cannot resolve a caller's reply queue, so the processor deleted the row and logged "Could not deserialize message body". The caller never received the fault and ran into its own timeout.

This PR routes faults through the reply dispatch endpoint like replies, in the outbox processor as well as the Postgres and in-memory scheduled message dispatchers, which share the same lookup. The processors now log the three drop causes separately: an unreadable envelope, an unknown message type and an unresolvable destination. An integration test reproduces the drop with a reply address outside the host's topology and verifies the caller receives the fault.

@github-actions github-actions Bot added 📚 documentation This issue is about working on our documentation. 🌶️ mocha labels Oct 12, 2026
@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Patch coverage

78.7% of changed lines covered (48/61)

File Covered Changed Patch %
…/Scheduling/ScheduledMessageDispatcher.cs 18 26 69.2% 🔴
…/Mocha.EntityFrameworkCore.Postgres/Outbox/OutboxProcessor.cs 21 26 80.8% 🟡
…/Outbox/PostgresMessageOutboxQueries.cs 4 4 100.0% 🟢
…/Scheduling/InMemoryScheduledMessageWorker.cs 2 2 100.0% 🟢
src/Mocha/src/Mocha/Headers/HeaderMessageKindExtensions.cs 3 3 100.0% 🟢
Uncovered changed lines (JSON)
{
  "sha": "0a79a89123d8fbcd25788e6f8a139e816492d17c",
  "files": [
    { "path": "src/Mocha/src/Mocha.EntityFrameworkCore.Postgres/Scheduling/ScheduledMessageDispatcher.cs", "ranges": [[156, 156], [158, 158], [174, 176], [240, 242]] },
    { "path": "src/Mocha/src/Mocha.EntityFrameworkCore.Postgres/Outbox/OutboxProcessor.cs", "ranges": [[168, 168], [170, 170], [239, 241]] }
  ]
}

Project coverage: 58.6% (315584/538115 lines)

@tobias-tengler tobias-tengler changed the title [Mocha] Fix the Postgres outbox dropping fault replies to a caller's reply queue [Mocha] Fix Postgres outbox dropping fault replies to a caller's reply queue Oct 12, 2026
@tobias-tengler
tobias-tengler marked this pull request as ready for review October 12, 2026 14:36
Copilot AI balanced review requested due to automatic review settings October 12, 2026 14:36

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The fix is focused and integration-tested, with only a minor naming issue remaining.

1 open finding
What changed in this PR

Fixes Postgres outbox delivery of fault replies targeting caller-owned reply queues.

Changes:

  • Routes replies and faults through reply dispatch endpoints.
  • Adds targeted Postgres outbox integration coverage.
  • Improves dropped-message diagnostics and scheduling documentation.
File Description
website/​content/​docs/​mocha/​scheduling.md Documents dropped-message causes.
PostgresOutboxFaultReplyTests.cs Tests fault reply delivery.
Helpers/​MessagingTestHost.cs Adds test host lifecycle support.
HeaderMessageKindExtensions.cs Detects replies and faults.
InMemoryScheduledMessageWorker.cs Routes faults through reply dispatch.
ScheduledMessageDispatcher.cs Improves routing and diagnostics.
OutboxProcessor.cs Fixes persisted fault reply routing.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

A fault reply to a caller whose reply queue is not a configured dispatch
endpoint is dropped by the Postgres outbox processor and the scheduled
message dispatcher. The tests reproduce the drop and pin the log messages
that name the cause.
The outbox processor and the scheduled message dispatcher only routed
MessageKind.Reply through the transport's reply dispatch endpoint. A
MessageKind.Fault reply took the generic dispatch lookup, which cannot
resolve a caller's reply queue, and was deleted with a "Could not
deserialize message body" log. Faults now take the reply dispatch
endpoint like replies, and the three drop causes are logged separately.
The tests started the runtime by hand and waited a fixed delay before
disposing the provider. MessagingTestHost starts the registered hosted
services in order and stops them in reverse, so the workers are stopped
before the provider is disposed. The scheduler test moves into its own
file on the same helper.
A reply or fault never reaches the scheduled message dispatcher through
the bus, since only publish, send and request set a scheduled time. The
test had to forge a scheduler row to reach that branch. The dispatcher
keeps routing faults like replies so it mirrors the outbox processor.
The Postgres outbox processor and scheduled message dispatcher deleted a row when
they could not deserialize its envelope, resolve its message type or resolve its
destination. During a rolling deployment an old instance could delete a message
written by a newer instance that only the newer instance understands.

Both processors now leave such a row in place with its incremented attempt count,
so the existing backoff retries it and another instance can send it. The skip is
logged as a warning. Once the row reaches the attempt limit it is no longer
selected and an exhausted entry is logged at critical level. The outbox lock
query returns the attempt count for that, and its limit is a named constant.

Also renames the reply check in the in-memory scheduled worker to say that it
covers faults as well.
@tobias-tengler
tobias-tengler force-pushed the tte/mocha-outbox-fault-reply branch from 20301d9 to ff0da33 Compare October 13, 2026 07:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

📚 documentation This issue is about working on our documentation. 🌶️ mocha

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants