[kibana-flaky-test-triage] On-demand workflow to triage failed-test issues by team

[kpollich]

Feature Idea

Summary: On-demand workflow that triages open failed-test issues in elastic/kibana for a specific team. Manually triggered (e.g. monthly), it queries all open flaky test issues for the given team, analyzes each one, and produces a single summary issue with categorized results and fix recommendations.

• Required input: team label (e.g. Team:Fleet). Never runs across all teams.
• For each issue: reads the structured issue body (test path, stack trace, buildkite link, codeowners), the test source file, and recent commits touching that file
• AI categorizes root cause and suggests specific fixes (e.g. "add retry to flaky assertion", "mock this HTTP endpoint")
• Output: single summary issue with categorized table of all open flaky tests, per-issue recommendations, and priority signal based on failure frequency
• Pilot with Team:Fleet, designed to work for any team label

Problem

• ~703 failed-test issues created or updated per week across Kibana
• Human triage is repetitive: read stack trace, check git blame, decide if flaky vs real
• Most failures fall into well-known patterns (timeouts, network dependencies, race conditions, real regressions)
• EMs have no aggregated view of their team's flaky test debt
• This workflow provides a periodic "state of flaky tests" report with actionable next steps per issue

Proposed Workflow

1. Trigger: Manual workflow dispatch with required team_label input (e.g. Team:Fleet)
2. Query: gh search issues --repo elastic/kibana --label failed-test --label $team_label --state open
3. Parse: For each issue, extract structured data from the issue body (test ID, file location, stack trace, codeowners, buildkite link)
4. Enrich: Read the test source file from elastic/kibana and check git log for recent commits touching the test file
5. Analyze: AI categorizes each issue (open-ended - not a fixed taxonomy) and recommends a specific fix
6. Report: Generate a single summary issue with:
   • Categorized table of all open flaky tests
   • Per-issue root cause analysis and recommended fix
   • Failure frequency to help prioritize

Implementation Considerations

• failed-test issue bodies are already well-structured with a metadata table, stack trace, and buildkite link (see evidence below)
• Test file paths are included in the issue body, making enrichment straightforward
• Read-only workflow - analysis and issue creation only, no code changes
• Scoped to one team at a time, keeping context manageable

Evidence

Example failed-test issues from Team:Fleet showing the structured format:

• Failing test: When the user has All privilege for Fleet but None for integrations - Fleet is accessible kibana#258970 - TimeoutError on page.waitForSelector (typical timeout flake)
• Failing test: X-Pack Fleet Package Policy API Integration Tests.x-pack/platform/test/fleet_api_integration/apis/package_policy/create·ts - Package policies Package Policy - create "before all" hook for "can only add to hosted agent policies using the force parameter" kibana#257844 - "before all" hook failure in package policy API test
• Failing test: X-Pack EPM API Integration Tests.x-pack/platform/test/fleet_api_integration/apis/epm/install_prerelease·ts - EPM Endpoints installs package that has a prerelease version should bulk install the GA packages when prerelease is not set kibana#257614 - EPM bulk install prerelease test failure

Each issue contains a structured table (Test ID, Target, Location, Duration, Module, Config path, Code Owners), a stack trace, and a Buildkite build link.

[github-actions]

tl;dr: The fastest path is to extend the existing flaky-test-investigator workflow into a manual, team-scoped Kibana variant (required team_label, target_owner, target_repo) instead of creating a net-new workflow from scratch.

Recommendation

Build kibana-flaky-test-triage as a focused variant of gh-aw-flaky-test-investigator, with required team and target-repo inputs, deterministic summary-issue dedupe, and explicit cross-repo token requirements.

Findings

• Existing workflow already does core triage mechanics (issue search, run/log inspection, frequency mapping, one issue output):
  • .github/workflows/gh-aw-flaky-test-investigator.md:94-109,125-133
• Current inputs are insufficient for this request (no team or target repo inputs):
  • .github/workflows/gh-aw-flaky-test-investigator.md:21-51
  • gh-agent-workflows/flaky-test-investigator/README.md:27-34
• Trigger is broader than requested (scheduled + workflow_run + manual):
  • gh-agent-workflows/flaky-test-investigator/example.yml:2-8
• Dedupe currently relies on title-prefix/fuzzy matching, not deterministic team-scoped key:
  • .github/workflows/gh-aw-flaky-test-investigator.md:75-80
  • .github/workflows/gh-aw-fragments/previous-findings.md:9-15,20-24
• Cross-repo querying pattern already exists in repo prompts (repo:{owner}/{repo}):
  • .github/workflows/internal-downstream-health.md:110-111
• Token setup today is repo-centric and must be documented for Kibana cross-repo read access:
  • scripts/quick-setup.sh:218-225,248
• Related prior work discovered:
  • Issue #153 “Add Flaky Test Triage”
  • PR #166 “Add flaky test triage workflow template”
    (full details were not retrievable in this run due integrity filtering)

Verification

I validated current behavior by inspecting the workflow source/docs and repository search outputs.

Command excerpts:

• rg "kibana|failed-test|flaky|triage" gh-agent-workflows .github/workflows
  • confirms existing flaky-test-investigator and related triage assets.
• python summary of GitHub search results
  • returned relevant prior items including #153 and #166.
• direct file inspection (view) of:
  • .github/workflows/gh-aw-flaky-test-investigator.md
  • .github/workflows/gh-aw-fragments/previous-findings.md
  • gh-agent-workflows/flaky-test-investigator/{README.md,example.yml}

No code changes were made in this triage run.

Detailed Action Plan

1. Add required workflow inputs for team/repo scoping in .github/workflows/gh-aw-flaky-test-investigator.md.

   • Add team-label (required), target-owner (default elastic), target-repo (default kibana) near existing inputs at :21-51.

2. Create a manual-only trigger example for this use case.

   • Add/update gh-agent-workflows/flaky-test-investigator/example.yml variant to workflow_dispatch only with required team_label input.

3. Update prompt data-gathering instructions to query Kibana failed-test issues by team.

   • In assignment section (:94+), require search filter equivalent to:
     • repo:(owner)/(repo) is:issue is:open label:failed-test label:"(team_label)"

4. Add structured field extraction requirements from issue body.

   • Parse and require: test path, stack trace snippet, Buildkite link, codeowners.

5. Add cross-repo enrichment instructions.

   • For extracted test paths, fetch source file + recent commits in target repo and include evidence links in summary.

6. Enforce deterministic dedupe key for “single summary issue”.

   • Replace title-only fuzzy duplicate logic (previous-findings fragment) with a stable marker (team+repo+window) in issue body or title convention.

7. Align docs with actual behavior.

   • Update gh-agent-workflows/flaky-test-investigator/README.md inputs/trigger/safe-output behavior to match implementation (currently says auto-close older triage reports while workflow has close-older-issues: false).

8. Document token prerequisites for cross-repo reads.

   • In README, explicitly require COPILOT_GITHUB_TOKEN to have access to elastic/kibana for issues/repos/actions reads.

Related Items

Type	Link	Relevance
Issue	#816	Requested feature (team-scoped Kibana flaky triage)
Issue	#153	Historical flaky triage request (search-discovered)
PR	#166	Historical flaky triage workflow template (search-discovered)
File	.github/workflows/gh-aw-flaky-test-investigator.md:21-51	Inputs currently missing team/repo scope
File	.github/workflows/gh-aw-flaky-test-investigator.md:94-109	Existing triage logic to reuse
File	.github/workflows/gh-aw-fragments/previous-findings.md:9-15	Current dedupe mechanism
File	gh-agent-workflows/flaky-test-investigator/example.yml:2-8	Trigger currently not manual-only
File	scripts/quick-setup.sh:218-225,248	Token setup context
File	.github/workflows/internal-downstream-health.md:110-111	Cross-repo search query precedent

Note

🔒 Integrity filtering filtered 5 items

Integrity filtering activated and filtered the following items during workflow execution.
This happens when a tool call accesses a resource that does not meet the required integrity or secrecy level of the workflow.

• issue:[kibana-flaky-test-triage] On-demand workflow to triage failed-test issues by team #816 (issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
• resource:search_pull_requests (search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
• issue:elastic/ai-github-actions#unknown (search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
• issue:Add Flaky Test Triage #153 (issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
• pr:Add flaky test triage workflow template #166 (pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)

---

What is this? | From workflow: Trigger Issue Triage

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.