CI wall clock is set by CIMarten and CIAzureServiceBus; compliance classes dominate transport suites

[jeremydmiller]

Where the CI wall clock actually goes

The tests.yml matrix runs ~30 jobs concurrently, so the workflow's wall clock is the slowest single job, not the total. From run 30634981923:

Job	Duration
CIMarten	18m
CIAzureServiceBus	18m
CISqlServer	14m
CIPubsub	12m
CIPersistence	12m
CIKafka	12m
CIRabbitMQ	11m
CIEfCore	11m
CIRedis	9m
CIRavenDb	9m
CICosmosDb	6m
CIPolecat / CICircuitBreaking / CIAWSSqs	5m
CIPulsar	3m

Two jobs set the number. Any work on a job below ~14m changes the workflow's wall clock by zero. That is worth stating explicitly because it is easy to optimise the suite you happen to be looking at.

What a measurement of one suite showed

Wolverine.Redis.Tests (144 tests) was measured in detail as a pilot — running it across worker processes, partitioned by test class:

Workers	Wall	Speed-up	Result
1	451.8s	1.00x	144/144
2	243.1s	1.86x	144/144
4	205.8s	2.20x	144/144
8	199.0s	2.27x	144/144

Green at every size, with retries disabled so nothing could mask instability. But it plateaus hard at ~199s, and the reason is the useful part:

Class	Total	Tests
BufferedSendingAndReceivingCompliance	188.1s	22
InlineSendingAndReceivingCompliance	178.6s	22
everything else (29 classes)	233s	100

Two compliance classes hold 61% of the suite's total test time (366.7s of 599.4s). Since a test class cannot be split across processes without breaking its isolation contract, the largest class is a hard floor: 599.4 / 188.1 = 3.19x is the ceiling, and no amount of parallelism goes below ~188s.

That floor is reachable from a single run — sum the per-test durations, divide by the largest class's total. It predicted 188s; the four-run sweep measured 199s. Within 6%. So the "how much would parallelism buy this job" question is answerable per suite without a bisection.

Why this probably generalises to the jobs that matter

BufferedSendingAndReceivingCompliance / InlineSendingAndReceivingCompliance are not Redis-specific — they derive from the shared Wolverine.ComplianceTests base, and the same classes exist across the transports:

• Wolverine.AzureServiceBus.Tests — 6 compliance classes (InlineSendingAndReceivingCompliance, BufferedSendingAndReceivingCompliance, sending_compliance_with_prefixes, two topic/subscription suites, …)
• Wolverine.RabbitMQ.Tests — 6 (durable_compliance, quorum_queue_compliance, stream_queue_compliance, …)
• Wolverine.Pubsub.Tests — 4
• Wolverine.Kafka.Tests, Wolverine.Nats.Tests — 1 each

CIAzureServiceBus is an 18m long pole and has six of them. If its time distribution looks like Redis's, it is both the most valuable job to attack and a better-shaped one — six large partitions parallelise further than two do.

This is unmeasured. The claim here is only that the shape is worth checking on CIAzureServiceBus and CIMarten first, because that is where the wall clock is.

Possible directions, roughly in order of cost

1. Measure CIAzureServiceBus and CIMarten the way Redis was measured: per-class time totals and the sum / largest-class ceiling. One run each. This decides whether anything below is worth doing.
2. Shard the long poles across CI jobs, the way CIPolecat / CIPolecatWorkflow / CIPolecatSagas already shard one project by namespace (CI: CIAWS and CIPolecat jobs chronically hit the 20-minute execution timeout #3350). Needs no new tooling, uses the existing testFilter parameter on RunTestProject, and turns one 18m job into several shorter concurrent ones. Most likely the cheapest real win.
3. Attack the compliance base itself. If two classes are 61% of a transport suite, and the same classes exist in eight transport suites, then shortening the compliance tests shortens many jobs at once. The slowest individual tests in the Redis run were 12–15s each (schedule_send, can_stop_receiving_when_too_busy_and_restart_listeners, can_schedule_retry) — worth checking whether those are inherently slow or waiting on fixed timeouts.
4. Multi-process parallelism within a job. Real (2.27x measured on Redis) but the largest-class floor limits it, and sharding across CI jobs gets much of the same benefit with machinery that already exists here.

Caveats on the numbers

• Measured on an Apple Silicon dev machine, not a CI runner. Absolute times will differ.
• The Redis suite is idle-dominated — the sequential run used 5% CPU over 7m38s — so it is waiting on brokers and polls, not computing. That is why parallelism helped despite limited cores, and it is a reason to expect similar behaviour on a CI runner. It may not hold for CPU-bound suites.
• ~37s of the Redis suite is unconditional Task.Delay (15s in ScheduledMessageTests, 6.5s in RetryLimitTests, 8s in DeadLetterQueueTests). Only ~8% of the run, so sleeps are not the main cost here — but they are the kind of cost that replacing with polling would remove and make more reliable.

[jeremydmiller]

Status after the 2026-08-01 session (all on #3758 unless noted):

• CIMarten: 18m → 9.5m (previous session, 2-lane clamp on hosted runners). Sibling-database lane-scoping and a possible memory-headroom question spun out to MartenTests sibling databases (tenant1-3, players, things) are shared across CI worker lanes #3762.
• CISqlServer: profiled 60 classes / 931s / largest 167s → 5.6× ceiling; runs at 4 workers (clamped to 2 on CI) with per-lane catalogs. 14m24s → 5m37s local; expect ~16m → ~9-10m on CI. Container boot now overlaps compilation (LaunchDockerServices/AwaitDockerServices).
• CIKafka: chronic red fixed on main (Kafka retry-tier listeners inherit native DLQ enablement; DLQ tests get isolated topics #3760) — shared-DLQ-topic pollution masking a real retry-tier DLQ bug.
• ASB flaky ledger: the 9-test group flake was the dispose-leak (thirteen new ValueTask DisposeAsync() no-op fixtures leaking hosts, fixed on Bobcat-supervised CI test runs: parallel workers, per-lane databases, honest retries #3758); retries-off went 228/239 → 238/239. Per-lane emulator design for the remaining sequential 17m — now the sole pole — is CIAzureServiceBus is the CI wall-clock pole: per-lane emulator parallelization #3761.
• Compliance-class dominance (this issue's second half) is unchanged: the compliance battery is what makes every broker suite's largest class the parallelism floor.

Remaining sequence: #3761 (ASB pole) → the ~12m broker tier (Kafka/Pubsub/RabbitMQ) with the same recipe → #3762 → the #3763 flaky-tag burn-down.

🤖 Generated with Claude Code

https://claude.ai/code/session_013eR4GL278688VhyhrGcttJ

[jeremydmiller]

Premise check against current data — half of this is now stale, and the other half moved.

Measured on the last green main run (30856898284), 30 jobs, 188 runner-minutes:

job	duration	% of 1200s cap
CIAzureServiceBus	1012s	84%
CIRabbitMQ	1023s	85%
CIPubsub	767s	64%
CISqlServer	751s	63%
CIPersistence	676s	56%
CIMarten	352s	29%

CIMarten is no longer a pole. The three-way namespace sharding did its job — it is now 29% of the cap and sits mid-pack, so the "CIMarten and CIAzureServiceBus" framing in the title no longer describes the run.

The current pair is CIAzureServiceBus and CIRabbitMQ, and CIRabbitMQ is the newer arrival — it went 749s → 1023s over the course of 2026-08-03. I have filed #3817 for that specifically, with the caveat that some of those runs were concurrent and its variance is high.

So the remaining work under this issue is really: ASB (tracked in #3790, whose blocking rationale has expired) and whatever is happening to CIRabbitMQ (#3817). Suggest retitling or narrowing this one accordingly.

[jeremydmiller]

Re-baselined against main run 30965904379 (99d5d0c65, green). The premise in the title no longer holds.

Top of the profile — 32 jobs, 203 runner-minutes, max 792s:

job	duration
CIRabbitMQ	792s
CISqlServer	781s
CIAzureServiceBus	775s
CIPubsub	762s
CIEfCore	689s
CIPersistence	679s
CIKafka	616s
…
CIMarten	352s (16th)

Two things changed since this was written:

1. CIMarten is no longer a pole. Namespace sharding took it to 352s — sixteenth place, under half the top job. This half of the title is simply stale.
2. There is no pole at all now. The top four are within 30 seconds of each other (792/781/775/762). That is the important part, and it changes what any follow-up work is worth: shrinking a single job no longer moves the wall clock, because three others immediately take its place. Wall clock is the max over jobs, so the only lever that pays is one that moves the whole leading group.

Concretely, this means proposals of the form "split job X" should be costed against ~10-30s of wall-clock gain, not against X's full duration. #3818 already demonstrated the cost side: sharding ASB three ways took the wall clock from 1012s to ~457s at the time, but added 2 jobs and 11 runner-minutes of fixed per-job overhead (checkout, build, container warm-up) that per-class measurement cannot see.

Suggest either retitling this to something like "No single CI wall-clock pole remains; the leading group is flat at ~790s" and rescoping to the compliance-class question that is still live, or closing it and opening a fresh issue against the real target. The compliance-classes-dominate-transport-suites observation may still be true and is worth keeping — it just is not a wall-clock argument any more.

One thing worth watching separately: CIAzureServiceBus (the catch-all shard) measured 775s here against 427s in the run #3818 landed on. That is one data point and could be variance, but the catch-all is where new classes land by design, so it is the shard most likely to drift.

[jeremydmiller]

Closing. Re-checked against the last two green main runs — 32642459117 (8/23, 21a5f5ab) and 32601687538 (8/22) — and the 8/5 verdict holds 18 days later.

35 jobs, 211 runner-minutes, max 806s:

job	8/23	8/22
CIRabbitMQ	806s	806s
CIEfCore	779s	770s
CIAzureServiceBus	770s	768s
CIPubsub	769s	785s
CISqlServer	726s	681s
CIMarten	385s	381s

Both halves of the title are now wrong. CIMarten is mid-pack at 32% of the 1200s cap. And the leading group is flat within 37 seconds — so no single-job split moves the wall clock, because three other jobs immediately take its place.

The compliance-class observation in the second half is still true and still unaddressed, but it is no longer a wall-clock argument: at 211 runner-minutes per run it is a runner-minutes argument, which is a different case that deserves its own issue and its own cost/benefit rather than living on under a stale title.

The two follow-ups this issue spawned are closed alongside it: #3761 (ASB, superseded by the #3818 shard, no drift observed) and #3817 (CIRabbitMQ, reverted to 806s).