Repository navigation
Refactored preliminary testing agent - #410
Conversation
Implement a new `preliminary-testing` supervisor workflow that automates the evaluation of preliminary testing (gating and CI checks) for RHEL Jira issues. Previously, QE engineers had to manually verify test results and set the Preliminary Testing field — this agent handles it automatically. The workflow uses an AI agent (BeeAI ToolCallingAgent) to analyze test results from two sources: 1. GreenWave gating status — fetches and interprets the HTML page from gating-status.osci.redhat.com for the build NVR when available. 2. OSCI results in MR comments — discovers linked merge requests via the Jira dev-status API, then fetches MR notes from GitLab to find OSCI "Results for pipeline ..." comments. The workflow gracefully degrades when only one source is available (e.g. no build NVR set yet, or no linked MRs). Entry conditions: issue must be In Progress, Preliminary Testing not already Pass, and at least one of Fixed in Build NVR or linked PRs. Outcomes: - Tests passed + Test Coverage set → sets Preliminary Testing = Pass - Tests passed + Test Coverage missing → flags for human attention - Tests failed/not running/error → flags for human attention - Tests running/pending → reschedules for later New files: - supervisor/preliminary_testing_handler.py — main workflow handler - supervisor/preliminary_testing_analyst.py — AI agent for analysis - supervisor/tools/fetch_greenwave.py — BeeAI tool for GreenWave HTML - supervisor/tools/fetch_gitlab_mr_notes.py — BeeAI tool for MR notes Modified files: - supervisor/jira_utils.py — add set_preliminary_testing() and get_issue_pull_requests() (Jira dev-status API) - supervisor/main.py — add preliminary-testing CLI command - Makefile — add preliminary-testing make target - README-supervisor.md — document the new workflow Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces the PreliminaryTestingAnalyst agent, which automates the analysis of RHEL JIRA issues to determine if preliminary testing and gating checks have passed. The implementation includes a new workflow-based agent, a standalone execution target in the Makefile, and several new tools for fetching GreenWave gating status, GitLab merge request notes, and JIRA development status. Feedback focuses on improving the robustness of the new tools by adding network timeouts to aiohttp sessions, implementing pagination for GitLab notes, and ensuring safer dictionary access. Additionally, there are suggestions to refine error messaging for JIRA component validation and to remove an unused parameter in the attention-flagging logic.
47dcc3e to
719e4d7
Compare
nforro
left a comment
nforro
left a comment
There was a problem hiding this comment.
LGTM, just a note, ToolCallingAgent is deprecated and we should avoid using it in new code.
719e4d7 to
1f38fd6
Compare
TomasTomecek
left a comment
TomasTomecek
left a comment
There was a problem hiding this comment.
Very nice work, Tome would you have time please to refactor the new tool into the existing one?
| issue_key: str = Field(description="Jira issue key (e.g. RHEL-12345)") | ||
|
|
||
|
|
||
| class GetJiraPullRequestsTool(Tool[GetJiraPullRequestsToolInput, ToolRunOptions, JSONToolOutput[list[dict[str, Any]]]]): |
There was a problem hiding this comment.
we already have a tool for this, at line 658
There was a problem hiding this comment.
here's Claude's analysis:
The overlap is substantial — they're essentially the same pattern.
Identical steps
- Resolve issue key → numeric ID via
rest/api/3/issue/{issue_key}?fields= - Fetch dev-status summary via
rest/dev-status/1.0/issue/summary?issueId={issue_id} - Loop over application types and fetch detail via
rest/dev-status/1.0/issue/detail?issueId={issue_id}&applicationType={app_type}&dataType=...
Key differences
Existing (get_jira_dev_status) |
New PR (GetJiraPullRequestsTool) |
|
|---|---|---|
dataType |
repository |
pullrequest |
| Summary key | summary.repository.byInstanceType |
summary.pullrequest.byInstanceType |
| Data extracted | commits (repositories[].commits[]) |
pull requests (pullRequests[]) |
| Output shape | {url, message, repository_url} |
full PR dict (id, name, status, url, source, destination, ...) |
| Error on detail fail | warns + continues | warns + continues |
| Session timeout | no explicit timeout | uses AIOHTTP_TIMEOUT |
Recommendation
The ID resolution + summary fetch is ~30 lines of logic duplicated verbatim. It would make sense to extract shared helpers — e.g. _resolve_issue_id() and _get_dev_status_summary() — to avoid maintaining two copies of the same HTTP calls.
The agent now conforms to practises used in the other agents and is located beside them in the agents subpackage.
1f38fd6 to
57fad7b
Compare
TomasTomecek
left a comment
TomasTomecek
left a comment
There was a problem hiding this comment.
LGTM, thank you for addressing the refactor
4 participants
This PR moves the agent created by mkyral beside other agents and modifies its implementation so it conforms to other agents that already exist.