Skip to content

Corrected test discovery with flaky retries#3073

Merged
jeremydmiller merged 6 commits into
JasperFx:mainfrom
dmytro-pryvedeniuk:corrent-nuke-test-discovery-5
Jun 11, 2026
Merged

Corrected test discovery with flaky retries#3073
jeremydmiller merged 6 commits into
JasperFx:mainfrom
dmytro-pryvedeniuk:corrent-nuke-test-discovery-5

Conversation

@dmytro-pryvedeniuk

@dmytro-pryvedeniuk dmytro-pryvedeniuk commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

This PR changes the way how the tests are executed.

  • Fix. Because of the custom test discovery some tests were ignored (e.g. ones that extend parent test class). Test discovery is fixed.
  • Flaky retries. Instead of using different scopes for test runs - per assembly, per class, per method - and retrying whole scope I run all tests per assembly (Round 1) and retry failed ones per method up to 2 times (Round 2). It provides uniform approach for all test projects, improves performance (no need to rerun successful tests) and protects better from flaky tests (existing or new).
  • Fix: CosmosDB and AzureServiceBus tests. New discovered tests required fixes to make runs green.
  • Marked some tests as flaky to skip for now. E.g. Pulsar compliance or RabbitMq CircuitBreaker tests. We need to address them separately.

Possible improvements:

  • For now flaky-during-run tests are just logged but later we can handle them better, e.g. report somehow or auto-create an issue.
  • If there are many failing tests, e.g. if DB is not available during run flaky retry approach can be slow. Some restriction can be added, e.g. no more than N failed tests to retry.

@dmytro-pryvedeniuk dmytro-pryvedeniuk force-pushed the corrent-nuke-test-discovery-5 branch from 7381698 to 02bb299 Compare June 11, 2026 12:40
@dmytro-pryvedeniuk dmytro-pryvedeniuk marked this pull request as draft June 11, 2026 12:43
@dmytro-pryvedeniuk dmytro-pryvedeniuk force-pushed the corrent-nuke-test-discovery-5 branch from 02bb299 to 0152d0c Compare June 11, 2026 12:46
@dmytro-pryvedeniuk dmytro-pryvedeniuk changed the title CI: Corrent nuke test discovery 5 Corrected test discovery with flaky retries Jun 11, 2026
@dmytro-pryvedeniuk dmytro-pryvedeniuk marked this pull request as ready for review June 11, 2026 13:35
@jeremydmiller

Copy link
Copy Markdown
Member

@dmytro-pryvedeniuk That's a big win! Thank you.

@jeremydmiller jeremydmiller merged commit 79934d7 into JasperFx:main Jun 11, 2026
24 checks passed
@dmytro-pryvedeniuk dmytro-pryvedeniuk deleted the corrent-nuke-test-discovery-5 branch June 11, 2026 20:05
This was referenced Jun 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants