Add RHOBS log collection to ROSA CI workflows - #83560
Conversation
Wire rosa-gather-rhobs-logs as a best_effort post step to the rosa-aws-sts-conformance and rosa-aws-sts-byo-kms-oidc workflows, and rosa-gather-hcp-diagnostics (which includes MC-side log collection) to the rosa-aws-hcp-conformance workflow. The gather steps are placed as the first post step in each workflow so the cluster is still available when logs are collected. Using best_effort: true ensures that a failure in log gathering does not fail the overall job. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository YAML (base), Central YAML (inherited) Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
Included review availability: Your plan includes up to 2 reviews per rolling hour; 0 remain after this review. WalkthroughThree ROSA workflow definitions now run best-effort post steps to collect HCP diagnostics or RHOBS logs. ChangesROSA post-processing
Estimated code review effort: 1 (Trivial) | ~5 minutes Merge Risk: ⚪ Minimal · up to This PR adds best-effort RHOBS log collection to three ROSA CI workflows without changing test, environment, or deprovision behavior; no actionable merge-blocking risk remains after normal checks and review. Suggested reviewers: Important Pre-merge checks failedPlease resolve all errors before merging. Addressing warnings is optional. ❌ Failed checks (1 error)
✅ Passed checks (14 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
[REHEARSALNOTIFIER]
A total of 26 jobs have been affected by this change. The above listing is non-exhaustive and limited to 25 jobs. A full list of affected jobs can be found here Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
|
/lgtm |
|
@dustman9000: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
openshift-merge-bot
Bot
added
the
rehearsals-ack
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: dustman9000, redhat-chai-bot The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
Bot
added
the
approved
|
@redhat-chai-bot: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
2 participants
Summary
Add RHOBS log collection (CS provisioning logs from RHOBS Loki) to three ROSA CI workflows that are currently missing it. This enables backend visibility into cluster install stall failures — specifically the OCM-to-Hive handoff failures tracked in ROSAENG-65470.
Problem
ROSA STS Classic e2e jobs (e.g.
e2e-rosa-sts-ovn) have a ~33% pass rate over the last 7 days, with ~65-70% of failures caused by cluster install stalls where:installingstate for 60+ minutesinfra_idremains null — Hive never creates the ClusterDeploymentprovision_error_codeor status description from OCMThe root cause investigation requires OCM cluster-service logs showing why the handoff to Hive failed, but these logs are not collected by the current CI workflows.
Solution
The
rosa-gather-rhobs-logsstep already exists in the step registry and can collect CS provisioning logs via the RHOBS Loki API. It queries theuhc-stagenamespace for log lines mentioning the cluster ID. However, it was only wired into therosa-e2e-hcpworkflow, not the STS Classic or HCP conformance workflows.This PR adds the gather step as a
best_effort: truepost step (first in the post chain, before deprovision) to three workflows:rosa-aws-sts-conformance— addsref: rosa-gather-rhobs-logsrosa-aws-sts-byo-kms-oidc— addsref: rosa-gather-rhobs-logsrosa-aws-hcp-conformance— addschain: rosa-gather-hcp-diagnostics(wrapsrosa-gather-rhobs-logsplus MC-side log collection)Impact
best_effort: trueensures the gather step does not fail the job if RHOBS credentials are missing or the query errorsMC_CLUSTER_IDfor Classic clusters (skips MC-side logs, collects CS logs only)AI-generated. Review for accuracy.
@dustman9000 requested in Slack thread
Summary by CodeRabbit