Test suites pollute each other through the shared 'wolverine' schema — a stale scheduled envelope from one transport's suite fails another's

[jeremydmiller]

Surfaced while working #3365. Not a product bug — a test-infrastructure one — but it manufactures phantom CI failures, which is the same tax #3350 is about.

What happened

On a clean run of the PolecatTests suite, try_to_use_the_session_transactional_middleware_end_to_end failed with:

UnknownTransportException: rabbitmq://queue/items

A Polecat test failing on a RabbitMQ URI. There is no RabbitMQ transport registered in that host, and nothing in the test references one.

Cause

A stale scheduled envelope left behind in the shared wolverine schema by a prior RabbitMQ test run. The durability agent in the Polecat host picked it up out of the shared envelope tables, tried to resolve its Destination — rabbitmq://queue/items — against a runtime with no RabbitMQ transport, and threw.

It drained on the next run and the test has passed since, which is exactly what makes this nasty: it is a non-deterministic, cross-suite, order-dependent failure that looks like a real defect in whichever suite happens to pick up the orphan.

Why it matters

• The failure lands in a suite that has nothing to do with the code that caused it, so the triage instinct ("what did I just change in Polecat?") points in the wrong direction.
• It is invisible on a re-run, so it reads as flake and gets ignored — which is also how a real intermittent bug gets ignored.
• It compounds the CI: CIAWS and CIPolecat jobs chronically hit the 20-minute execution timeout #3350 problem: we already can't tell a genuine CI failure from a job timeout; this adds a third category (someone else's leftover row).

Directions to consider

• Isolate the schema per suite. The cleanest fix — each test assembly gets its own schema rather than sharing wolverine. Costs some setup time.
• Purge scheduled/outgoing envelopes on suite setup rather than trusting the previous run to have drained cleanly.
• Make an unresolvable Destination non-fatal for the durability agent — arguably the durability agent should log-and-dead-letter an envelope whose transport it cannot resolve rather than throwing, since that is also a plausible production shape (a node rolls out without a transport that a queued envelope still references). This one may be a real product question hiding behind a test annoyance, and is the part I find most interesting.

The third bullet is worth deciding on its own merits even if we fix the isolation.

Refs #3350, #3365.