Make flaky detector verify recurrence postdates the fix before commenting - #13930
Merged
Conversation
…ting The detector's closed-issue handling commented 'recurred' on any related issue closed within ~30 days, without checking whether the failures actually occurred after the fix merged. Because the look-back window (-DaysBack) routinely spans the fix date, stale pre-fix failures were mislabeled as recurrences (e.g. issue #13734, fixed the day before). Step 4 now requires confirming at least one failure's build start time is strictly after the fixing PR merged (using the closing PR's merge commit time, and build start times rather than the date-only lastSeen) before posting a recurrence comment; otherwise the evidence is treated as stale and the test is skipped for the run. SKILL.md gains a matching note. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
🔍 Skill Validator Results
Summary
Full validator output```text Found 1 skill(s) [flaky-test-detector] 📊 flaky-test-detector: 3,581 BPE tokens [chars/4: 3,475] (standard ~), 13 sections, 5 code blocks [flaky-test-detector] ⚠ Skill is 3,581 BPE tokens (chars/4 estimate: 3,475) — approaching "comprehensive" range where gains diminish. ✅ All checks passed (1 skill(s)) ``` |
Contributor
There was a problem hiding this comment.
Pull request overview
Updates the flaky-test detector’s guidance for handling recently-closed tracking issues so it only comments “recurred” when there is evidence of failures occurring after the fix PR merged, avoiding stale pre-fix failures being misclassified as recurrences.
Changes:
- Adds a timing-aware recurrence check for closed issues: require at least one post-fix failure (by build start time) before commenting.
- Documents the same rule in the flaky-test-detector skill’s “Interpreting Results” guidance.
Show a summary per file
| File | Description |
|---|---|
| .github/workflows/flaky-test-detector.agent.md | Adds post-fix timestamp verification steps before commenting on recently-closed issues. |
| .github/skills/flaky-test-detector/SKILL.md | Documents the post-fix recurrence rule in the skill guidance. |
Copilot's findings
- Files reviewed: 2/2 changed files
- Comments generated: 1
gh issue view --json does not support a timelineItems field. Use the supported closedByPullRequestsReferences field to get the closing PR, then gh pr view --json mergedAt for its merge time. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
AlesProkop
approved these changes
Jun 3, 2026
Member
|
@ViktorHofer pls squash next time |
This was referenced Jun 4, 2026
This was referenced Aug 11, 2026
Bump Microsoft.Build from 18.4.0 to 18.9.6
SkylineCommunications/Skyline.DataMiner.CICD.Packages#172
Open
This was referenced Aug 19, 2026
Closed
Open
This was referenced Aug 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes the flaky-test detector's closed-issue handling timing-aware so it stops mislabeling stale, pre-fix failures as recurrences.
Problem
When the detector finds a flaky test that maps to a recently-closed tracking issue, it commented "recurred" on that issue without checking when the failures occurred. Because the look-back window (
-DaysBack) routinely spans the date a fix merged, failures from before the fix were treated as evidence of recurrence. For example, issue #13734 was fixed via a merged PR the day before, yet the detector commented that it had recurred based on pre-fix builds.Change
.github/workflows/flaky-test-detector.agent.md(Step 4): before posting a recurrence comment on a closed issue, the agent must now confirm that at least one failure's build start time is strictly after the fixing PR merged (using the closing PR's merge-commit time and actual build start times, not the date-onlylastSeen). If every failure predates the fix/close, the evidence is treated as stale: no comment, no reopen, skip the test for the run..github/skills/flaky-test-detector/SKILL.md: companion note in the "Interpreting Results" section documenting the same rule.Body/prose-only edits to the agent workflow and skill — no lock recompile required.