Add AI analysis to medik8s jobs - #82802
Conversation
Signed-off-by: Alejandro Brugarolas <abrugaro@redhat.com>
openshift-ci
Bot
added
do-not-merge/work-in-progress
|
Hi @abrugaro. Thanks for your PR. I'm waiting for a openshift member to verify that this patch is reasonable to test. If it is, they should reply with Regular contributors should join the org to skip this step. Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughThe PR adds a Claude-based Medik8s E2E failure-analysis step. It detects failed test steps, generates analysis artifacts, and runs after ChangesMedik8s E2E failure analysis
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant ProwJob
participant GCSArtifacts
participant ClaudeCLI
participant SharedDirectory
ProwJob->>GCSArtifacts: Poll finished.json files
GCSArtifacts-->>ProwJob: Return test status
ProwJob->>ClaudeCLI: Run Prow failure-analysis skill
ClaudeCLI-->>ProwJob: Stream analysis and usage metrics
ProwJob->>SharedDirectory: Write reports and token data
Suggested labels: Suggested reviewers: Important Pre-merge checks failedPlease resolve all errors before merging. Addressing warnings is optional. ❌ Failed checks (1 error)
✅ Passed checks (14 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
/pj-rehearse pull-ci-medik8s-fence-agents-remediation-main-5.0-openshift-e2e |
|
@abrugaro: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
@abrugaro: needs-ok-to-test label found, no rehearsals will be run |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@ci-operator/step-registry/medik8s/analyze-e2e-failure/medik8s-analyze-e2e-failure-commands.sh`:
- Around line 33-44: Update the TEST_STEPS loop in the analysis polling flow so
a passed step does not immediately exit via the success branch. Track each
step’s passed status, continue scanning all configured steps for failures, and
exit successfully only after every step reports .passed == true; preserve
immediate failure handling when any step reports .passed == false.
- Around line 73-101: Restrict the Claude invocation in the analysis command to
a credential-isolated environment: remove credential-bearing variables such as
GOOGLE_APPLICATION_CREDENTIALS from the agent context, prevent Bash and WebFetch
from accessing or exfiltrating credentials, and allow only the artifact download
and report-writing operations required by the SYSTEM_PROMPT. Update the tool
configuration around the claude command while preserving its existing fast
analysis and artifact output behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Pro Plus
Run ID: 01a6336e-0eb2-48ca-8775-3c563873e823
📒 Files selected for processing (8)
ci-operator/config/medik8s/fence-agents-remediation/medik8s-fence-agents-remediation-main__5.0.yamlci-operator/config/medik8s/self-node-remediation/medik8s-self-node-remediation-main__4.22.yamlci-operator/config/medik8s/self-node-remediation/medik8s-self-node-remediation-main__4.23.yamlci-operator/config/medik8s/self-node-remediation/medik8s-self-node-remediation-main__5.0.yamlci-operator/step-registry/medik8s/analyze-e2e-failure/OWNERSci-operator/step-registry/medik8s/analyze-e2e-failure/medik8s-analyze-e2e-failure-commands.shci-operator/step-registry/medik8s/analyze-e2e-failure/medik8s-analyze-e2e-failure-ref.metadata.jsonci-operator/step-registry/medik8s/analyze-e2e-failure/medik8s-analyze-e2e-failure-ref.yaml
|
/ok-to-test |
openshift-ci
Bot
added
ok-to-test
|
/pj-rehearse pull-ci-medik8s-fence-agents-remediation-main-5.0-openshift-e2e |
|
@abrugaro: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
@abrugaro oooh.. thank you :) waiting on results :) hopefully we have a failed test to observe |
|
/pj-rehearse pull-ci-medik8s-self-node-remediation-main-4.22-openshift-e2e |
|
@abrugaro: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
@abrugaro doesn't seem like the ai ananlysis executed. Perhaps because the build log is in a diff location than original script. /me looks. |
Signed-off-by: Alejandro Brugarolas <abrugaro@redhat.com>
|
/pj-rehearse pull-ci-medik8s-self-node-remediation-main-4.22-openshift-e2e |
|
@abrugaro: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
Signed-off-by: Alejandro Brugarolas <abrugaro@redhat.com>
|
/pj-rehearse pull-ci-medik8s-self-node-remediation-main-4.22-openshift-e2e |
|
@abrugaro: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse pull-ci-medik8s-fence-agents-remediation-main-5.0-openshift-e2e |
|
@abrugaro: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse pull-ci-medik8s-self-node-remediation-main-4.23-openshift-e2e |
|
@abrugaro: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
I would prefer to see this in action for one operator instead of merging this change for all, even though I don't see any risk. I guess it is just me being pessimistic about AI post-review. |
@razo7 which operator would you like to begin with? I'll leave only that one and add the rest once we see it working as exected. |
I think NHC/SBR/SNR since we have some upcoming/open features to test for them |
Signed-off-by: Alejandro Brugarolas <abrugaro@redhat.com>
|
/pj-rehearse pull-ci-medik8s-node-healthcheck-operator-main-4.22-openshift-e2e |
|
/pj-rehearse pull-ci-medik8s-node-healthcheck-operator-main-4.23-openshift-e2e |
|
@abrugaro: your |
1 similar comment
|
@abrugaro: your |
|
[REHEARSALNOTIFIER]
Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
|
/pj-rehearse pull-ci-medik8s-node-healthcheck-operator-main-4.22-openshift-e2e |
|
@abrugaro: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse pull-ci-medik8s-node-healthcheck-operator-main-5.0-openshift-e2e |
|
@abrugaro: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse ack |
|
@razo7: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
openshift-merge-bot
Bot
added
the
rehearsals-ack
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: abrugaro, razo7 The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
Bot
added
the
approved
* add AI analysis to medik8s jobs Signed-off-by: Alejandro Brugarolas <abrugaro@redhat.com> * add claude-ai-helper to job images Signed-off-by: Alejandro Brugarolas <abrugaro@redhat.com> * add forced failure Signed-off-by: Alejandro Brugarolas <abrugaro@redhat.com> * add forced failure Signed-off-by: Alejandro Brugarolas <abrugaro@redhat.com> * add ai analysis to all operators Signed-off-by: Alejandro Brugarolas <abrugaro@redhat.com> * leave nhc only Signed-off-by: Alejandro Brugarolas <abrugaro@redhat.com> --------- Signed-off-by: Alejandro Brugarolas <abrugaro@redhat.com>
3 participants
Add
medik8s-analyze-e2e-failurestep registry ref that uses Claude Code (via the pre-installedai-helpersCI plugin) to automatically analyze E2E test failures for medik8s operators
Runs as a
best_effortpost-step: polls GCS forfinished.json, invokes the/ci:prow-job-analysisskill when failures are detected, and writes a markdown analysis report tothe artifact directory
For now wired only into NHC (4.22, 4.23, 5.0)
Forced an error to see the analysis in prow, see it here
Here is an example of a whole analysis: failure-analysis-sbr-77.md
Summary by CodeRabbit
Adds the
medik8s-analyze-e2e-failureCI step for Medik8s operator E2E workflows.The step polls GCS for test results, analyzes failures with Claude Code and the
/ci:prow-job-analyze-test-failureskill, and writes a Markdown report to the artifact directory. It runs as a best-effort post-step with a 30-minute timeout.The step is enabled for Fence Agents Remediation, Self Node Remediation, Machine Deletion Remediation, Node Healthcheck Operator, Node Maintenance Operator, and Storage-Based Remediation across their 4.22, 4.23, and 5.0 configurations.