CI shard hosts are dying and taking random tests down with them — 4 different tests failed across 4 runs of inert diffs while main stayed green #1242

Activity

1. rbuergi commented on Aug 12, 2026

   

   rbuergi

   on Aug 12, 2026

   ContributorAuthor

   More actions

   Still happening 2026-08-12 — and now it is reddening main itself, four times in one morning

   The original report noted "main was green throughout". That is no longer true. Four post-merge runs on main failed today, every one carrying the host-death signature and each naming a different test in a different assembly:

   time (UTC)	run	merge	failing test	shard
   07:04	31572467193	#1258	Hosting.Monolith.Test.DynamicTypePreWarmerTest.RegressedType_ThatRebuildsAfterTheContentConverges_IsRetracted	2
   07:20	31573558342	#1262	FutuRe.Test.FutuReAnalysisTest.EuropeRe_AnnualReport_EmbeddedCharts_ShouldRenderViaPathResolution	2
   07:35	31574636901	#1255	AI.Test.SkillAutocompleteTest.Autocomplete_ListsBuiltInSkills_FromCatalog	3
   08:21	31577940567	#1260	ContentCollections.Test.NodeHubContentCollectionTest.CollectionNamedArea_RendersBrowserForFolder_AndContentFo…	2

   All four annotate as:

   Shard N: a test host exited non-zero (failure, crash, or timeout kill)
   A test host exited non-zero despite the trx gate (crashed or killed mid-run — completed-only results were published)
   

   Two things this adds to the original report:

   1. It is no longer PR-only. A red main is a different severity: the "compare against green main" move that currently lets an author prove innocence stops working when main itself is intermittently red.
   2. Shard 2 in three of four. The original table spread across shards 1–4, so this may be noise, but it is the first hint of clustering and worth checking against runner sizing before assuming uniformity.

   ⚠️ A counter-example from the same morning — do not let this issue become a blanket excuse

   PR #1256 is a markdown-only diff that reddened shards 3 and 4 in AI.Test — superficially an exact match for this issue, and I called it a flake and re-ran it. The re-run failed identically, which is what forced a real look:

   BuiltInSkillCatalogTest.EveryShippedSkillFile_LoadsAsASkillNode
   BuiltInSkillCatalogTest.SkillCatalog_LoadsEveryShippedFile_WithNoSkippedFile
   SkillMarkdownRoundTripTest.EveryBuiltInSkill_RoundTrips_SerializeThenParse_Lossless
   

   It was a genuine regression: the edited skill's front-matter description contained an unquoted : (…authoring a card: three rules…), so the YAML no longer parsed and the skill stopped loading. Verified both directions locally — with the colon exactly those three tests fail, without it all 8 pass.

   The discriminators that separate the two, in the order they are cheap to check:

   • Same tests twice ⇒ real. This issue names different victims each run; a repeat of the identical set is a bug.
   • a test host exited non-zero ⇒ this issue. A named assertion with Failed: N in the trx and no host-death line ⇒ real.
   • "markdown-only, therefore inert" is FALSE in this repo. content/ai/** ships as loadable skill nodes with tests over them, so a docs-shaped diff genuinely can break AI.Test. That assumption is what cost the extra cycle here.

   Worth folding into the issue body, since "inert diff" is the report's central premise and it has at least one real exception.
2. rbuergi

   closed this as completedin #1268on Aug 12, 2026