Bug Hunter Improvements

[strawgate]

See https://github.com/elastic/beats/actions/runs/22206903339

We already know tests are passing, so why is it running tests?

The goal is to identify a bug / logic issue / etc

[github-actions]

TL;DR: Bug Hunter is not explicitly told to run full test suites, but the current prompt strongly nudges it toward test-driven repro (minimal failing test), so it can spend time re-running tests even when CI is already green; I recommend tightening the prompt to prefer targeted repro and explicitly avoid redundant full-suite runs.

Recommendation

Update .github/workflows/gh-aw-bug-hunter.md to keep the reproducibility bar high while reducing unnecessary test execution:

1. Keep mandatory reproduction, but explicitly prefer the smallest targeted command and avoid rerunning full test suites that are already green unless needed for a new failing signal.
2. Relax wording from “must be covered by a minimal failing test” to “must have deterministic reproduction (failing check or clearly incorrect output/behavior).”
3. Add an explicit skip rule for “re-running broad existing test suites with no new failure hypothesis.”

Findings

• Current Bug Hunter prompt emphasizes test framing:
  • .github/workflows/gh-aw-bug-hunter.md:79 — requires a bug that can be covered by a minimal failing test.
  • .github/workflows/gh-aw-bug-hunter.md:101 — “Can be expressed as a minimal failing test.”
  • .github/workflows/gh-aw-bug-hunter.md:92-95 — reproduction is mandatory (good, should stay).
• There is no explicit instruction today to skip redundant full-suite reruns when status is already known/green.
• Related historical context:
  • #89 and PR #90 focused on duplicate prevention/labeling, not test-execution strategy.
  • #158 clarified commit-history lookback behavior.
  • #135 added “run required commands” guidance for code-changing workflows; Bug Hunter is detection/reporting and should still optimize for focused repro over broad reruns.

Verification

$ rg -n "test|tests|passing|failing" .github/workflows/gh-aw-bug-hunter.md
79: Find a single reproducible, user-impacting bug ... minimal failing test.
101: - Can be expressed as a minimal failing test ...
136: > [File path + outline of test]

$ web_fetch https://github.com/elastic/beats/actions/runs/22206903339
Status Success; artifact list available from run page summary.

I could confirm prompt wording locally; the linked run page summary is accessible, but detailed command logs are not included in that summary fetch.

Detailed Action Plan

1. Edit .github/workflows/gh-aw-bug-hunter.md:
   • At :79, reword goal to emphasize deterministic repro with minimal targeted commands (not broad suite reruns).
   • In Data Gathering step 3 (:92-95), add a bullet: if CI/tests are already green for the same scope, do not rerun full suites unless there is a concrete failure hypothesis.
   • In “What to Look For” (:99-102), replace strict “minimal failing test” language with “deterministic failing check or clear behavior mismatch.”
   • In “What to Skip” (:103-109), add “rerunning broad test suites without a new hypothesis.”
2. Recompile workflow locks (make compile) so .github/workflows/gh-aw-bug-hunter.lock.yml reflects prompt changes.
3. Run lint/validation (make lint) and spot-check the compiled lock includes the new guidance text.
4. Observe 2-3 subsequent Bug Hunter runs to verify behavior shifts toward focused repro and fewer redundant test reruns.

Related Items

Type	Link	Relevance
Issue	#185	Current request
File	.github/workflows/gh-aw-bug-hunter.md:79,92-95,99-109,136	Prompt text driving Bug Hunter behavior
Issue	#89	Prior duplicate-prevention improvements
PR	#90	Implemented #89 prompt updates
Issue	#158	Prior scope clarification (recent commits)
Issue	#135	“Run required commands” guidance for code-changing workflows
External Run	https://github.com/elastic/beats/actions/runs/22206903339	Referenced run showing concern

---

What is this? | From workflow: Issue Triage Agent

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not. for issue #185

[strawgate]

I want it to find a bug, build a test that shows that the bug exists, etc. It's not supposed to be just running tests hoping one fails and then calling that a bug