CI/testing sweep: compliance assertion scoping, Marten segmentation, dispose leaks

[jeremydmiller]

Addresses the open testing/CI issues, plus the two red jobs on main today. Closes #3726, #3729, #3771, #3762, #3751, #3725; partially addresses #3763.

The two findings that were not what the issues said

#3729 — closed as a "genuine 40% flake"; it is a test bug with a traced mechanism

multi_node_exclusive_listener_recovery.rows_released_after_the_listener_is_already_running_are_still_recovered seeded its rows owned by a fabricated node number (987654) standing in for a departed node. A fabricated owner is an orphan, and freeing orphans is a job the product does on purpose:

-- ReleaseOrphanedMessagesOperation, on the durability agent's schedule
update wolverine_incoming_envelopes set owner_id = 0
where owner_id != 0 and owner_id not in (select node_number from wolverine_nodes)

So the rows stopped being "owned by another node" on their own, the exclusive listener's 250ms recovery poll then claimed them perfectly correctly, and

tracking.Count.ShouldBe(0,
    "The listener must not touch inbox rows that are still owned by another node");

failed on intended behaviour. Instrumenting the 2-second window shows the order directly — on a failing run:

DIAG t=800ms  globallyOwned=0  tracked=0
DIAG t=900ms  globallyOwned=5  tracked=0     <- orphan release
DIAG t=1400ms globallyOwned=0  tracked=5     <- listener, correctly

ListenerInboxRecovery only ever loads owner_id = 0 rows, so the listener never touched a row owned by anyone. The product is right; the test's premise was wrong.

Fixed by seeding from another live node — live rows are not orphans, so nothing is entitled to free them, and the assertion becomes a real statement about the listener. (InboxStaleTime, the other thing that bumps owned rows, is null unless explicitly configured and this test does not set it.)

Verified: 12/12 consecutive passes against a 3-in-8 baseline (P(chance) ≈ 0.35%), plus the full class 3×.

#3726 — the three tests it names are all already resolved

Verified locally: item 1 passes (fixed by merged #3728), item 3 passes (that was #3729), and item 2 was only ever a missing local SQL Server container. A full CIRabbitMQ --disable-test-retry run was 435 passed / 1 failed, and the one failure was #3729 above.

That run surfaced the current chronic failure instead — which is what took CI red today.

quorum_queue_compliance — today's red CIRabbitMQ

The transport compliance error-handling helpers picked their "ending activity" by message type:

session.AllRecordsInOrder()
    .Where(x => x.Envelope!.Message is ErrorCausingMessage)
    .LastOrDefault(... MessageSucceeded or MovedToErrorQueue ...)

Broker queues are shared across the test methods in a compliance class, so a redelivery still in flight when one test's tracked session completed lands in the next test's session and becomes its ending record. Today's log shows it directly: message 08def019-82b1-43da arrives at 27ms in one test and 28ms in another — before either had sent anything — under two different node ids, and its MessageSucceeded at 3913ms is what both assertions latched onto.

ErrorCausingMessage now carries an Id and the three helpers scope to it. That is immune to residue by construction; purging cannot help, because an unacked in-flight redelivery is not in the queue to purge.

#3771 / #3762 — CIMarten runner deaths

CIMarten was killed outright by the runner twice in a row at ~15 minutes on an unrelated diff — the OOM signature, not a test failure. The worker-lane split is the cause: several Marten hosts plus Postgres does not fit a 4-vCPU/16GB hosted runner. The build already recorded that 4 lanes had killed a runner once; the clamp to 2 still died.

Lanes are gone. The suite is segmented across three CI jobs the way CIPolecat is (#3350), each strictly sequential against exactly one database — concurrency moves off the runner VM and onto the matrix, where every shard gets its own 16GB.

Job	Namespaces	Size
CIMartenWorkflow	AggregateHandlerWorkflow, Bugs, ScheduledJobs, Requirements	54 classes / 162 tests
CIMartenSagas	Saga, Persistence (incl. Persistence.Sagas), Distribution, MultiTenancy, AncillaryStores, Dcb	60 classes / 205 tests
CIMarten	everything else + MartenSubscriptionTests	46 classes / 190 tests

Verified disjoint and exhaustive against a real MartenTests --list-tests (159 classes / 545 tests). The catch-all is the negation of the two named shards, so a brand-new namespace lands somewhere instead of being silently dropped. (The root namespace can never be an include-list entry — every display name starts with it — which is exactly why the catch-all has to be the negation.)

This also closes #3762: with a single lane per job the sibling databases (tenant1-3, players, things) have one owner again, so no lane-scoping sweep is needed.

A memory sampler stays on the three Marten jobs to confirm the footprint actually dropped. It prints to the live job log on purpose — when the runner dies, later steps are skipped, so an artifact upload would never run.

#3763 — the public new DisposeAsync() shape

34 compliance fixtures declared public new DisposeAsync(). It reads like an override and is not one: TransportCompliance<T> disposes through a T-typed reference, which binds statically to the base. Every one of them was dead code.

• 7 carried real cleanup that had therefore never executed — a SQLite database, the shared-memory queues, and four LocalMqttBrokers, each leaking a listening socket for the life of the test process. Moved to AfterDisposeAsync (virtual, and actually runs).
• 4 were infinitely self-recursive (await DisposeAsync() calling itself).
• The rest were redundant and are deleted.

The trap is now documented on the base method. This is the same shape #3758 fixed for Azure Service Bus; that sweep just never covered the other transports.

#3751 — Testcontainers fixtures leak one container per process

The Redis/MQTT/Mqtt5/Pulsar [ModuleInitializer] fixtures never disposed their container, relying entirely on Ryuk. With ryuk.disabled=true every test process leaks one permanently, and the process count is rising. Took the small-diff option from the issue — a ProcessExit hook that keeps the static-property ergonomics — rather than the invasive IAsyncLifetime conversion. Each also honours a WOLVERINE_* connection string when set, so a caller can supply a warm broker and skip a start that sits before Main on every process launch.

#3725 — finish the xUnit1051 rollout

TracingTests opted in (clean, no diagnostics), the Directory.Build.targets opt-in block deleted, and the now-redundant property dropped from all 72 projects. Full wolverine.slnx builds clean with the analyzer enforced everywhere.

Verification

• Full wolverine.slnx builds clean (0 warnings, 0 errors).
• LightweightTcpTransportCompliance 22/22 — exercises all four changed error-handling assertions and proves the new message Id round-trips through serialize/transport/deserialize.
• All six RabbitMQ compliance classes green (136 tests).
• CIRabbitMQ --disable-test-retry: 435/436, the one failure being multi_node_exclusive_listener_recovery: listener touches inbox rows still owned by a departed node (~40% failure) #3729, now fixed.

Two things I could not verify locally, called out honestly:

1. The quorum poisoning does not reproduce locally — three baseline runs on main passed on a fast machine with a fresh broker. It needs CI's slower, loaded conditions. CIRabbitMQ on this PR is the real verification.
2. Finish the xUnit1051 rollout: convert TracingTests, then delete the opt-in block #3725 asks that TracingTests be run, not just compiled. It compiles clean with the analyzer enforced; CIOtel here is the outstanding evidence.

Not addressed: the #3763 Category=Flaky ledger itself (~50 tests), and #3752/#3761 (CIAzureServiceBus wall clock).

🤖 Generated with Claude Code

https://claude.ai/code/session_013eR4GL278688VhyhrGcttJ