CI throughput: 211 runner-minutes per run makes concurrent merges unverifiable for hours

[jeremydmiller]

Split out of #3752, which was closed because its premise — "CIMarten and CIAzureServiceBus set the wall clock" — had gone stale. One observation in it did not go stale, and it needs a title that describes the real constraint rather than a wall-clock one.

Wall clock is fine. Runner-minutes are the constraint.

Measured on the last fully-completed main run, 32649119089 (c3e1c24d7, 34/34 green):

• 35 jobs, 211 runner-minutes, longest job 827s, median job 350s.

The 20-minute cap is comfortable — the longest job sits at 69% of it — and there is no pole: the top four are within 66 seconds of each other. Optimising any single job still buys ~nothing in wall clock, exactly as #3752 concluded.

But every merge costs 211 runner-minutes, and that is what bit today.

What happened on 2026-08-23

Five merges landed in about an hour (#4016, #4020, #4021, #4023, #4024, then #4025 and #4027). The queue reached:

• 26 runs queued, 0 in progress, for roughly two and a half hours.
• Of the queued runs, 11 were tests.yml — about 2,300 runner-minutes of backlog.

Consequence: three merges to main were unverifiable for hours. main was confirmed green only through c3e1c24d7; the runs covering #4024, #4025 and #4027 had not started a single job. When a change merges before its own matrix finishes — which happened with #4024 — that verification gap is the only thing standing behind it.

This is a different failure mode from the one the CI effort has been optimising. Latency per run is healthy; throughput under concurrent merges is not, and no issue tracks it.

Where the minutes actually go

Same run, jobs by duration:

job	s		job	s
CIRabbitMQ	827		CIRedis	584
CISqlServer	790		CIPulsar	477
CIAzureServiceBus	763		CIAzureServiceBusRouting	460
CIPubsub	761		CIRavenDb	443
CIPersistence	699		CIAzureServiceBusLeader	422
CIEfCore	644		CIPolecat	413
CIKafka	615		…21 more, 15s–384s

The distribution is flat — no single job is more than 7% of the total — so there is nothing here to shard. Sharding moves minutes between jobs and adds them (#3818 added 11 runner-minutes of fixed per-job overhead: checkout, build, container warm-up). Against a runner-minutes budget, sharding is a cost, not a win. That inverts the recommendation that made sense when wall clock was the target.

The lead worth measuring

#3752's surviving observation: the shared compliance battery dominates the transport suites. Measured on Wolverine.Redis.Tests, two compliance classes held 61% of the suite's total test time (366.7s of 599.4s), and those same classes exist across eight transport suites. If that generalises, the compliance battery is a large fraction of the 211 minutes — and unlike sharding, shortening it removes minutes outright, from many jobs at once.

Also from that measurement: the Redis suite used 5% CPU over 7m38s. It is idle-dominated — waiting on brokers and polls — and ~37s of it was unconditional Task.Delay. Fixed sleeps are the kind of cost that polling would remove and make more reliable.

Suggested first step — measure, do not act

The per-class profile that exists was taken to answer a wall-clock question. Retake it against runner-minutes:

1. For the top ~8 jobs by duration, sum per-class test time from a TRX and rank by total minutes contributed, not by "does this class set the floor".
2. Report what fraction of the 211 minutes is the shared compliance battery, and what fraction is unconditional sleeps.
3. Only then decide whether to shorten the battery, replace sleeps with polling, or accept the cost and raise concurrency instead.

Nothing here should be acted on before that measurement. The last time this was acted on from the wrong framing, it produced a sharding change that helped wall clock and hurt runner-minutes.

Explicitly not proposed

• Sharding anything further. See above.
• Cutting coverage. The compliance battery is what makes the transports comparable; the question is whether it costs what it should, not whether to run it.

[jeremydmiller]

Profiled it. The lead in the issue body is wrong, and there is no hotspot to attack.

Method: run 32651915002 (01e04aab6, green, 34 CI jobs, 210.5 runner-minutes). Job durations from the API; phase boundaries parsed from the 34 job logs by timestamp. Real runner data, not a laptop.

Where the 210.5 minutes go

phase	min	share
test execution	152.9	72.7%
nuke bootstrap + restore + compile	22.7	10.8%
container readiness	13.3	6.3%
teardown + ledger upload	4.7	2.2%
runner boot + checkout + setup-dotnet	2.6	1.2%
unattributed (step transitions)	~14.1	6.7%

Fixed per-job cost — everything that is not test execution or container wait — is ~53s, paid 34 times, ~30 minutes total (14%). That is the number to cost sharding against: every new shard buys another ~53s plus its own container warm-up, before running a single test.

The compliance battery is not the problem

The issue body carried #3752's lead forward: two compliance classes were 61% of Wolverine.Redis.Tests' time, the same classes exist in eight suites, therefore the shared battery is a large fraction of the total. It does not hold cross-suite.

TransportCompliance<T> contributes 22 tests per derived class, across 46 derived classes — 1,078 of 6,060 tests, 18% by count. Splitting the jobs by how much of each is compliance-derived:

group	tests	test-minutes	s/test
jobs ≥25% compliance	1,919	48.2	1.51
jobs <25% compliance	4,141	104.7	1.52

No premium at all — 1.0x. Compliance-heavy suites cost exactly the median per test. Shortening the battery would remove ~18% of the tests and about ~18% of the time, which is a normal-sized win for a large amount of work, not the concentrated one the lead implied. The Redis result was real but local to Redis; it does not generalise, and I should not have carried it into this issue as a lead without checking that.

Sleeps are not the problem either

Census of literal Task.Delay/Thread.Sleep across all test projects: 236 sites, 341s nominal for a single execution of each — 5.7 minutes. Compliance-base sleeps multiply per derived class (the // I FEEL DIRTY Task.Delay(5.Seconds()) in TransportCompliance.cs:339 runs once per derived class = 46 × 5s = 3.8 min on its own), so the realistic total is roughly 9–10 minutes, ~6% of test execution. Worth fixing for reliability; irrelevant to throughput.

What the data does show

Per-test cost is remarkably flat: 6,060 tests, median 1.46 s/test. Only three jobs exceed 3× the median, and together they account for just 12.5 minutes of excess (8% of test time):

job	tests	s/test	excess over median
CIAzureServiceBusLeader	21	13.94	4.4 min
CIAzureServiceBusRouting	27	11.97	4.7 min
CICircuitBreaking	24	10.06	3.4 min

Those are leader-election and circuit-breaker suites — inherently timing-bound, and small. Fixing all three perfectly saves ~6% of the total.

Conclusion: the 153 minutes is 6,060 genuinely broker- and database-bound tests at ~1.5s each, spread evenly. There is no 20% sitting in one place. Anyone hoping to halve this by optimising a suite should stop here.

What that leaves

Two structural levers, neither of which is "optimise a suite":

1. Don't run the matrix when it cannot tell you anything. GH-4033: skip the test matrix for markdown-only changes #4036 did this for markdown. The same reasoning extends to other path classes — a change confined to one transport does not need the other twelve. That is a real design question (correctness of the filter vs. minutes saved), but it is the only lever with a large number behind it.
2. Attack the ~30 minutes of fixed per-job cost, which is 14% and pure overhead: restore + compile is 22.7 of it. Caching the build across matrix jobs, or building once and fanning out artifacts, would cut most of it. Note this points the opposite way from sharding — fewer, larger jobs are cheaper in runner-minutes, and each shard added since has been paying 53s for the privilege.

I would not act on the compliance battery or the sleeps for throughput reasons. Both are defensible on other grounds (reliability, suite clarity); neither moves this number.

[jeremydmiller]

I just flat out don't want to destabilize what we already have, so I'm closing this.