Skip to content

Fix flaky mutants#2717

Merged
martincostello merged 1 commit into
mainfrom
fix-flaky-mutant
Aug 22, 2025
Merged

Fix flaky mutants#2717
martincostello merged 1 commit into
mainfrom
fix-flaky-mutant

Conversation

@martincostello

Copy link
Copy Markdown
Member

Fix flaky mutants after task refactoring in #2664.

Fix flaky mutants after task refactoring in #2664.
Copilot AI review requested due to automatic review settings August 22, 2025 14:50

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR addresses flaky mutants in the Polly.Core circuit breaker controller by adding Stryker mutation testing exclusions. The changes disable mutation testing for specific conditional checks that are difficult to test reliably.

  • Added Stryker disable comments for two conditional checks
  • Prevents flaky mutation test failures for code paths that are hard to test deterministically

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

Comment thread src/Polly.Core/CircuitBreaker/Controller/CircuitStateController.cs
Comment thread src/Polly.Core/CircuitBreaker/Controller/CircuitStateController.cs
@codecov

codecov Bot commented Aug 22, 2025

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.12%. Comparing base (0159145) to head (fc6b235).
⚠️ Report is 1 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #2717   +/-   ##
=======================================
  Coverage   96.12%   96.12%           
=======================================
  Files         309      309           
  Lines        7118     7118           
  Branches     1008     1008           
=======================================
  Hits         6842     6842           
  Misses        222      222           
  Partials       54       54           
Flag Coverage Δ
linux 96.12% <ø> (ø)
macos 96.12% <ø> (ø)
windows 96.11% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@pentp

pentp commented Aug 22, 2025

Copy link
Copy Markdown
Contributor

Isn't the Debug.Assert(task.IsCompleted) assert below those conditions enough to fix the mutation tests? IIRC mutation tests always run in debug mode and those asserts should ensure that the opposite condition will fail, no?
If that's not the case, then some of the other Debug.Asserts I added might also be flaky.

@martincostello

Copy link
Copy Markdown
Member Author

They are, but I've seen two different failing mutations since the merge.

"configuration": "Debug",

@pentp

pentp commented Aug 22, 2025

Copy link
Copy Markdown
Contributor

Oh, I think the reason is because Debug.Assert also gets mutated (removed) and if done together with the other mutation, then both will "survive". The reason I thought asserts would be enough is that in our codebase we have "ignore-methods": [ "Assert" ] in Stryker config and I had forgotten it's not the default.

@pentp

pentp commented Aug 22, 2025

Copy link
Copy Markdown
Contributor

This condition might also be flaky then:

if (!whenAnyHedgedTask.IsCompleted)

Though I'm surprised that the similar condition below for if (completedTask == delayTask) doesn't seem to be flaky...

@martincostello

Copy link
Copy Markdown
Member Author

I'd be happy to accept a PR to tweak our configuration to do the same and then revert these changes so the conditions are mutated again.

I'm going to merge this now though as 4-in-a-row failures in main and 3-in-a-row successes here suggests the flaky CI is at least resolved for now (🤞).

@martincostello martincostello merged commit 651508b into main Aug 22, 2025
40 checks passed
@martincostello martincostello deleted the fix-flaky-mutant branch August 22, 2025 15:19
@pentp

pentp commented Aug 22, 2025

Copy link
Copy Markdown
Contributor

It's actually in the config already, so now I don't know why it was flaky:

"Debug.Assert",

@martincostello

Copy link
Copy Markdown
Member Author

Just had another one happen here:

if (!whenAnyHedgedTask.IsCompleted)

LogicalNotExpression to un-LogicalNotExpression mutation Survived (132:13)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants