Flaky test report: committed-code failures on 2026-04-01

[andrross]

Flaky test report: committed-code failures on 2026-04-01

Summary

11 test failures were detected across committed-code builds (Timer and Post Merge Action) in the past 24 hours. These map to 10 distinct failing tests (capped at 10 for this report). One test (ConcurrentSeqNoVersioningIT) reproduced locally with the original seed.

Failing Tests

1. ClientYamlTestSuiteIT — string profiler via global ordinals

• Build: 73664
• Seed: 2F166AF4E6F1B4F1
• Error: field [profile.shards.0.aggregations.0.debug.segments_with_single_valued_ords] is not greater than [0]
• Reproduced locally: No
• First seen: 2024-03-25
• Total unique builds affected: 350
• Pattern: Persistent long-running flake since March 2024. Major spike in Sep 2024 (89 builds). Steady at 4–24 builds/month since. Stable/chronic.

2. MixedClusterClientYamlTestSuiteIT — string profiler via global ordinals

• Build: 73660
• Seed: 6D4A15A66DCB92CF
• Error: Same as Bump com.diffplug.spotless from 5.6.1 to 6.2.0 #1 — segments_with_single_valued_ords not greater than 0
• Reproduced locally: Skipped (BWC test requires old cluster versions)
• First seen: 2024-03-25
• Total unique builds affected: 185
• Pattern: Same root cause as Bump com.diffplug.spotless from 5.6.1 to 6.2.0 #1 but in the mixed-cluster variant. Spike in Sep 2024 (83 builds). Lower frequency than Bump com.diffplug.spotless from 5.6.1 to 6.2.0 #1. Stable/chronic.

3. AwarenessAllocationIT — testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness

• Build: 73640
• Seed: D2E4E2BABCE53B0C
• Error: Expected: <120> but: was <118>
• Reproduced locally: Inconclusive (Gradle infra error)
• First seen: 2024-08-31
• Total unique builds affected: 135
• Pattern: Quiet until Apr 2025, then consistent 3–18 builds/month. Trending upward. Worsening.

4. RecoveryWhileUnderLoadIT — testRecoverWhileUnderLoadAllocateReplicasRelocatePrimariesTest

• Build: 73633
• Seed: E6978E901AAC624
• Error: NoSuchElementException: No value present
• Reproduced locally: Inconclusive (Gradle infra error)
• First seen: 2024-05-03
• Total unique builds affected: 117
• Pattern: Sporadic early on, major spikes in Apr–Jul 2025 (21–38 builds/month). Resurgence in Feb–Mar 2026 (8–11 builds/month). Worsening.

5. FullRollingRestartIT — testFullRollingRestart_withNoRecoveryPayloadAndSource

• Build: 73652
• Seed: 6E21A05A37A8A2E3
• Error: replica shards haven't caught up with primary expected:<18> but was:<15>
• Reproduced locally: No
• First seen: 2024-10-11
• Total unique builds affected: 104
• Pattern: Dormant until Jul 2025, then massive spike (47 builds). Quieted, then resurgence in Feb–Mar 2026 (17–13 builds/month). Worsening.

6. AzureBlobStoreRepositoryTests — testWriteRead

• Build: 73707
• Seed: None available
• Error: RepositoryVerificationException: path not accessible on cluster-manager node (BlobAlreadyExists/CONFLICT)
• Reproduced locally: Skipped (requires Docker for Azure fixture)
• First seen: 2024-04-29
• Total unique builds affected: 79
• Pattern: Persistent low-level flake since Apr 2024. Consistent 1–9 builds/month. Slight upward trend recently. Stable/chronic.

7. ConcurrentSeqNoVersioningIT — testSeqNoCASLinearizability

• Build: 73690
• Seed: C23880DF2C94CBC3
• Error: ClusterHealthResponse has timed out
• Reproduced locally: Yes — failure reproduced deterministically with seed C23880DF2C94CBC3
• First seen: 2024-10-03
• Total unique builds affected: 66
• Pattern: Consistent 1–10 builds/month since Oct 2024. Spike in Feb 2026 (10 builds). Stable/chronic.

8. RestoreShallowSnapshotV2IT — testRestoreShallowSnapshotIndexAfterSnapshot

• Build: 73696
• Seed: BD4077D675BFCA75
• Error: timed out waiting for green state
• Reproduced locally: No
• First seen: 2024-09-03
• Total unique builds affected: 50
• Pattern: Consistent 1–8 builds/month since Sep 2024. Quiet Mar–May 2025, then resumed. Stable/chronic.

9. TelemetryMetricsEnabledSanityIT — testGauge

• Build: 73653
• Seed: FA09294D2AF3FDAA
• Error: expected:<2.0> but was:<3.0>
• Reproduced locally: No
• First seen: 2024-07-15
• Total unique builds affected: 30
• Pattern: Low frequency until Mar 2026, then sudden spike to 17 builds in a single month. Worsening.

10. WarmIndexSegmentReplicationIT — testPrimaryStopped_ReplicaPromoted

• Build: 73668
• Seed: 8644DC5BA7FFD3B5
• Error: RefreshFailedEngineException: Refresh failed (CorruptIndexException: misplaced codec footer)
• Reproduced locally: Inconclusive (Gradle infra error)
• First seen: 2025-03-14
• Total unique builds affected: 16
• Pattern: Sporadic, 1–4 builds/month with gaps. Stable/low frequency.

Summary Table

#	Test	Builds Affected	First Seen	Trend	Reproduced
1	ClientYamlTestSuiteIT string profiler via global ordinals	350	2024-03	Stable/chronic	No
2	MixedClusterClientYamlTestSuiteIT string profiler via global ordinals	185	2024-03	Stable/chronic	Skipped
3	AwarenessAllocationIT testThreeZone...LoadAwareness	135	2024-08	Worsening	Inconclusive
4	RecoveryWhileUnderLoadIT testRecoverWhileUnderLoad...	117	2024-05	Worsening	Inconclusive
5	FullRollingRestartIT testFullRollingRestart...	104	2024-10	Worsening	No
6	AzureBlobStoreRepositoryTests testWriteRead	79	2024-04	Stable/chronic	Skipped
7	ConcurrentSeqNoVersioningIT testSeqNoCASLinearizability	66	2024-10	Stable/chronic	Yes
8	RestoreShallowSnapshotV2IT testRestoreShallowSnapshot...	50	2024-09	Stable/chronic	No
9	TelemetryMetricsEnabledSanityIT testGauge	30	2024-07	Worsening	No
10	WarmIndexSegmentReplicationIT testPrimaryStopped_ReplicaPromoted	16	2025-03	Stable/low freq	Inconclusive

Notes

• WarmIndexSegmentReplicationIT.testReplicaHasDiffFilesThanPrimary (build 73690, 9 unique builds affected, first seen 2025-03) was the 11th distinct failure and excluded from the top 10.
• Tests marked "Inconclusive" failed due to Gradle test infrastructure errors (missing output files), not test assertion failures.
• Tests marked "Skipped" could not be run locally due to environment requirements (BWC cluster versions, Docker).
• The string profiler via global ordinals failures (Bump com.diffplug.spotless from 5.6.1 to 6.2.0 #1 and Bump com.diffplug.spotless from 5.6.1 to 6.2.1 #2) share the same root cause — a YAML REST test assertion that depends on segment topology.