Flaky test report: committed-code failures on 2026-06-16

[andrross]

Flaky test report: committed-code failures on 2026-06-16

Investigation window: UTC 2026-06-15 10:00 → 2026-06-16 10:00 (last 24 hours)

Source: opensearch-metrics cluster, gradle-check-* index. Filter: test_status=FAILED AND ((invoke_type=Timer AND git_reference=main) OR (invoke_type=Post Merge Action AND pull_request_title starts with "push trigger main")).

11 distinct failing tests were observed across 9 affected gradle-check builds (10 reproduced, 1 class-level setup failure not separately reproduced).

Reproduction caveats

• All reproductions ran on the dev desktop (m6a-class instance, 16 cores), which is faster than the gradle-check m7a.8xlarge runners on most workloads but with a different CPU. Concurrency-window flakes can reproduce at different rates.
• Build JDK: Eclipse Temurin 21.0.10. Tests ran on OpenSearch 3.8.0-SNAPSHOT at commit 90fdbf95a0fbb22514004d7f21747fb9411446f3. The CI builds were against various commits in the same range; minor code drift between local HEAD and the failing build's commit is possible but small (all within the last 24h on main).
• Each test was run once with the original seed. The flaky-test playbook recommends ≥10 iterations to claim a seed is non-deterministic; a single passing run is suggestive but not conclusive. Reported as "no repro" below.
• The local working tree has a few uncommitted modifications under server/src/main/java/org/opensearch/common/util/concurrent and server/src/main/java/org/opensearch/threadpool/. None of the failed tests under investigation directly exercise those classes, but a broad behavioral effect cannot be ruled out.

Summary table (sorted by unique builds historically affected)

#	Test	Module	Recent build	Repro w/ seed	First seen	Unique builds	Pattern
1	RemoteStoreStatsIT.testNonZeroPrimaryStatsOnNewlyCreatedIndexWithZeroDocs	server (IT)	80487	no	2024-03-27	219	worsening (recent6m=54 vs prior6m=20)
2	RestoreShallowSnapshotV2IT.classMethod (class-level setup/teardown)	server (IT)	80475	not attempted	2025-05-26	113	stable (recent6m=60 vs prior6m=44)
3	SharedClusterSnapshotRestoreIT.testSnapshotFileFailureDuringSnapshot	server (IT)	80459	no	2024-08-31	112	stable (recent6m=37 vs prior6m=27)
4	RestoreShallowSnapshotV2IT.testHashedPrefixTranslogMetadataCombination {p0=writable_warm:true}	server (IT)	80475	no	2024-11-28	80	stable (recent6m=31 vs prior6m=37)
5	ClusterShardLimitIT.testOpenIndexOverLimit {p0=writable_warm:true}	server (IT)	80599	no	2025-10-15	76	worsening (recent6m=62 vs prior6m=14)
6	SecureReactorNetty4HttpServerTransportTests.testBadRequest	plugins:transport-reactor-netty4 (unit)	80515	yes	2025-02-19	69	improving (recent6m=4 vs prior6m=46)
7	DataFormatAwareEngineTests.testApplyMergeChangesUpdatesCatalogAndNotifiesListeners	server (unit)	80506	no	2026-05-08	32	worsening (recent6m=32 vs prior6m=0) — newly flaky
8	EquivalenceIT.testDuelDepthBreadthFirst {p0=concurrent_segment_search.mode:auto}	server (IT)	80459	no	2024-05-03	26	stable, sporadic
9	MergedSegmentWarmerIT.testCleanupRedundantPendingMergeFile	server (IT)	80570	no	2025-07-31	11	stable, low rate
10	IngestFromKafkaIT.testKafkaIngestionWithFieldMappingMapper	plugins:ingestion-kafka (IT)	80461	no	2026-04-28	6	worsening — new flake (started 2026-04)
11	IngestFromKafkaIT.testMultiThreadedWrites	plugins:ingestion-kafka (IT)	80455	no	2025-05-10	6	stable, low rate

Reproduction summary: 1 of 10 attempted tests reproduced deterministically with the original seed (SecureReactorNetty4HttpServerTransportTests.testBadRequest). The remaining 9 passed locally on a single seeded run, consistent with non-deterministic flakes whose seed only controls Random streams and not thread scheduling, network timing, or GC pauses.

---

Per-test details

1. RemoteStoreStatsIT.testNonZeroPrimaryStatsOnNewlyCreatedIndexWithZeroDocs — 219 builds since 2024-03

• Recent failure: build 80487 (Timer, main)
• Seed: D163626594AFA0D4:1C94B7B9D2B07712 — did not reproduce locally on a single run.
• Pattern: Long-running chronic flake. Highest historical volume in the report (219 affected builds across 27 months). Recent 6-month rate (54 builds) is meaningfully higher than the prior 6 months (20 builds). After a quieter 2025 mid-year, rate has been climbing again from ~2/mo in Q4 2025 to ~14/mo in 2026-05 and 2026-06.
• Categorization: Worsening chronic flake.

2024-03: 1   2024-04: 8   2024-05:14   2024-06: 5   2024-07:13   2024-08:19
2024-09:19   2024-10:16   2024-11: 9   2024-12: 8   2025-01:16   2025-02: 5
2025-03: 3   2025-04: 2   2025-05: 4   2025-06: 3   2025-07: 3   2025-08: 7
2025-09: 3   2025-10: 2   2025-11: 3   2025-12: 2   2026-01: 7   2026-02: 2
2026-03: 9   2026-04: 8   2026-05:14   2026-06:14

2. RestoreShallowSnapshotV2IT.classMethod (class-level) — 113 builds since 2025-05

• Recent failure: build 80475 (Timer, main)
• Seed: 89EEA578EB925503 — class-level setup/teardown failure, not separately reproduced (the per-method failure for this class is row 4 below).
• Pattern: This classMethod record represents @BeforeClass/@AfterClass failures in RestoreShallowSnapshotV2IT. It is consistently among the highest-volume failures: 8–14 builds/month with no obvious trend.
• Categorization: Stable, high-rate chronic flake. Likely indicative of teardown leakage from preceding test methods in the same class (i.e. row 4 leaving the shared cluster in a bad state).

3. SharedClusterSnapshotRestoreIT.testSnapshotFileFailureDuringSnapshot — 112 builds since 2024-08

• Recent failure: build 80459 (Post Merge Action)
• Seed: 5C1C044FA5A1FF28:636EA8255C9C9935 — did not reproduce locally on a single run.
• Pattern: Chronic flake at ~3–10 builds/month. Slight uptick in 2026 Q2 but within historical range.
• Categorization: Stable chronic flake.

4. RestoreShallowSnapshotV2IT.testHashedPrefixTranslogMetadataCombination {p0=writable_warm_index.enabled:true} — 80 builds since 2024-11

• Recent failure: build 80475 (Timer, main)
• Seed: 89EEA578EB925503 — did not reproduce locally on a single run.
• Pattern: Chronic flake with a slight downtrend from a 2025-Q3 peak (10 builds in 2025-07). Consistently 2–8 builds/month.
• Categorization: Stable chronic flake. Closely correlated with the class-level failure in row 2 (same class, same builds).

5. ClusterShardLimitIT.testOpenIndexOverLimit {p0=writable_warm_index.enabled:true} — 76 builds since 2025-10

• Recent failure: build 80599 (Timer, main)
• Seed: 64F821A351719C87:35575113DF8B1BF1 — did not reproduce locally; both parameter variants (writable_warm_index.enabled=true and =false) ran cleanly.
• Recent assertion: AssertionError: CloseIndexResponse failed - not acked at ClusterShardLimitIT.java:667.
• Pattern: First seen 2025-10. Steady at 6–9 builds/month through early 2026, then jumped to 19 in 2026-05 and 16 in 2026-06.
• Categorization: Worsening flake; the recent 2x rate increase is a fresh signal.

6. SecureReactorNetty4HttpServerTransportTests.testBadRequest — 69 builds since 2025-02

• Recent failure: build 80515 (Post Merge Action)
• Seed: FB220F2BBC10CF05:EA134488FD6FBBFE — REPRODUCED LOCALLY with the original seed on the first run, with the same exception:
  • reactor.core.Exceptions$ReactiveException: reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response
  • Stack trace identical to the Jenkins failure (originating at SecureReactorNetty4HttpServerTransportTests.testBadRequest:342).
• Pattern: Big spike of 33 builds in 2025-09, then went silent for 5 months (2026-01 through 2026-05 had zero failures). Resurfaced in 2026-06 with 4 builds. The seed-deterministic behavior plus the resurfacing pattern is consistent with an environmental sensitivity that this seed exposes specifically.
• Categorization: Long-term improving overall, but the deterministic reproducer is the most actionable lead in this report. Worth filing/linking to a focused issue.

7. DataFormatAwareEngineTests.testApplyMergeChangesUpdatesCatalogAndNotifiesListeners — 32 builds since 2026-05

• Recent failure: build 80506 (Timer, main)
• Seed: 43A2DE490EA817A4:30697DD1DA458521 — did not reproduce locally on a single run.
• Pattern: Brand-new flake. Zero history before 2026-05-08; then 28 builds in 2026-05 alone.
• Categorization: Newly-introduced flake. The high rate so soon after the test (or its subject) was added strongly suggests either a real bug in newly-added code or a test that was never stabilized. Highest priority candidate among the new flakes.

8. EquivalenceIT.testDuelDepthBreadthFirst {p0=concurrent_segment_search.mode:auto} — 26 builds since 2024-05

• Recent failure: build 80459 (Post Merge Action)
• Seed: 5C1C044FA5A1FF28:97F03C066422E5BA — did not reproduce locally on a single run.
• Pattern: Sporadic over a long timeframe. Long quiet stretches followed by clusters of 2–4 failures in a month.
• Categorization: Low-rate stable flake.

9. MergedSegmentWarmerIT.testCleanupRedundantPendingMergeFile — 11 builds since 2025-07

• Recent failure: build 80570 (Timer, main)
• Seed: 1CF50A877EEC367C:A544575D4923BF2E — did not reproduce locally on a single run.
• Pattern: ~1 build/month, occasional gaps of 2–3 months.
• Categorization: Low-rate stable flake.

10. IngestFromKafkaIT.testKafkaIngestionWithFieldMappingMapper — 6 builds since 2026-04

• Recent failure: build 80461 (Post Merge Action)
• Seed: 8729F13BAE321119:724116F79F933ED4 — did not reproduce locally on a single run.
• Pattern: New flake from 2026-04. Rate climbing slowly: 1 → 2 → 3 builds/month.
• Categorization: Newly introduced flake; trend warrants monitoring.

11. IngestFromKafkaIT.testMultiThreadedWrites — 6 builds since 2025-05

• Recent failure: build 80455 (Timer, main)
• Seed: CC7DBFE615BDF2E5:868F91C4AACA6F31 — did not reproduce locally on a single run.
• Pattern: 3 builds in 2025-05, then dormant for ~12 months, then 1 in each of 2026-05 and 2026-06.
• Categorization: Low-rate sporadic; not currently noisy but recently active.

---

Recommendations (no code changes proposed in this report)

1. SecureReactorNetty4HttpServerTransportTests.testBadRequest — Has a deterministic reproducer with the failing seed. This is the only test in this batch where the next investigator can step into a debugger directly.
2. DataFormatAwareEngineTests.testApplyMergeChangesUpdatesCatalogAndNotifiesListeners — Brand new and high-rate (32 builds in ~5 weeks). Worth investigating before it becomes chronic.
3. ClusterShardLimitIT.testOpenIndexOverLimit — 2x rate increase in the last two months. Worth checking whether a recent change to the writable-warm-index feature flag path is responsible.
4. RemoteStoreStatsIT.testNonZeroPrimaryStatsOnNewlyCreatedIndexWithZeroDocs — Chronic with a recent uptick. If an [AUTOCUT] issue does not already exist, one should be opened.
5. The two RestoreShallowSnapshotV2IT rows (per-method + class-level) are the same underlying class; address together.

Artifacts

Local logs and raw data are kept under /tmp/flaky-report-2026-06-16/:

• failures.json — list of distinct failures observed in the 24h window
• extracted-failures.json — Jenkins error details and seeds per failure
• historical.json — monthly aggregations from the metrics cluster
• report-data.json — the structured data behind the table above
• repro-*.log — full gradle output for each reproduction attempt