OTA-1814: fix(alerts): Remove duplicate information in the recommend command - #2279
Merged
openshift-merge-bot[bot] merged 2 commits intoJul 21, 2026
Merged
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughThe alerts function now detects when the ClusterUpdateAcceptRisks feature gate is enabled and hypershift is not in use; under those conditions, it returns early to skip client-side alert evaluation. Two helper functions inspect cluster FeatureGate and Infrastructure state to determine these conditions, and new unit tests validate the helper behavior. ChangesConditional alert checking based on feature gates and hypershift
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes 🚥 Pre-merge checks | ✅ 13 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (13 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
wking
reviewed
Jun 5, 2026
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@pkg/cli/admin/upgrade/recommend/recommend.go`:
- Around line 187-190: The call to o.Client.ConfigV1().FeatureGates().Get(ctx,
"cluster", metav1.GetOptions{}) is ignoring its error return; capture the error
(e.g., err) instead of using `_` and log it with klog.V(4).Infof() (including
context like that the FeatureGate fetch failed) while keeping the existing nil
fallback behavior for featureGate; update the block that references featureGate
and o.Client to handle the captured error and log it appropriately.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 260e5e72-5faa-490c-859c-5290a98e1dc7
📒 Files selected for processing (1)
pkg/cli/admin/upgrade/recommend/recommend.go
hongkailiu
reviewed
Jun 11, 2026
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
pkg/cli/admin/upgrade/recommend/recommend_test.go (1)
34-67: ⚡ Quick winExpand test coverage with table-driven tests.
These tests only cover the positive cases. Critical edge cases are missing: nil inputs, version mismatches, and feature-not-enabled scenarios. Table-driven tests would improve coverage and align with guidelines.
As per coding guidelines: "Unit tests should use co-located
*_test.gofiles with table-driven tests."♻️ Proposed table-driven tests
-func TestIsAcceptRisksEnabled(t *testing.T) { - featureGate := &configv1.FeatureGate{ - Status: configv1.FeatureGateStatus{ - FeatureGates: []configv1.FeatureGateDetails{ - { - Version: "4.22.0", - Enabled: []configv1.FeatureGateAttributes{ - { - Name: features.FeatureGateClusterUpdateAcceptRisks, - }, - }, - }, - }, - }, - } - - result := isAcceptRisksEnabled(featureGate, "4.22.0") - if result != true { - t.Errorf("ClusterUpdateAcceptRisks feature gate should report as enabled") - } -} - -func TestIsHypershiftEnabled(t *testing.T) { - infra := &configv1.Infrastructure{ - Status: configv1.InfrastructureStatus{ - ControlPlaneTopology: configv1.ExternalTopologyMode, - }, - } - - result := isHypershiftEnabled(infra) - if result != true { - t.Errorf("Hypershift should report as enabled") - } -} +func TestIsAcceptRisksEnabled(t *testing.T) { + tests := []struct { + name string + featureGate *configv1.FeatureGate + clusterVersion string + want bool + }{ + { + name: "nil feature gate", + featureGate: nil, + clusterVersion: "4.22.0", + want: false, + }, + { + name: "feature gate enabled for matching version", + featureGate: &configv1.FeatureGate{ + Status: configv1.FeatureGateStatus{ + FeatureGates: []configv1.FeatureGateDetails{ + { + Version: "4.22.0", + Enabled: []configv1.FeatureGateAttributes{ + {Name: features.FeatureGateClusterUpdateAcceptRisks}, + }, + }, + }, + }, + }, + clusterVersion: "4.22.0", + want: true, + }, + { + name: "feature gate enabled for different version", + featureGate: &configv1.FeatureGate{ + Status: configv1.FeatureGateStatus{ + FeatureGates: []configv1.FeatureGateDetails{ + { + Version: "4.21.0", + Enabled: []configv1.FeatureGateAttributes{ + {Name: features.FeatureGateClusterUpdateAcceptRisks}, + }, + }, + }, + }, + }, + clusterVersion: "4.22.0", + want: false, + }, + { + name: "feature gate not in enabled list", + featureGate: &configv1.FeatureGate{ + Status: configv1.FeatureGateStatus{ + FeatureGates: []configv1.FeatureGateDetails{ + { + Version: "4.22.0", + Enabled: []configv1.FeatureGateAttributes{ + {Name: "SomeOtherFeature"}, + }, + }, + }, + }, + }, + clusterVersion: "4.22.0", + want: false, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got := isAcceptRisksEnabled(tt.featureGate, tt.clusterVersion) + if got != tt.want { + t.Errorf("isAcceptRisksEnabled() = %v, want %v", got, tt.want) + } + }) + } +} + +func TestIsHypershiftEnabled(t *testing.T) { + tests := []struct { + name string + infra *configv1.Infrastructure + want bool + }{ + { + name: "nil infrastructure", + infra: nil, + want: false, + }, + { + name: "external topology mode (hypershift)", + infra: &configv1.Infrastructure{ + Status: configv1.InfrastructureStatus{ + ControlPlaneTopology: configv1.ExternalTopologyMode, + }, + }, + want: true, + }, + { + name: "high availability mode", + infra: &configv1.Infrastructure{ + Status: configv1.InfrastructureStatus{ + ControlPlaneTopology: configv1.HighlyAvailableTopologyMode, + }, + }, + want: false, + }, + { + name: "single replica mode", + infra: &configv1.Infrastructure{ + Status: configv1.InfrastructureStatus{ + ControlPlaneTopology: configv1.SingleReplicaTopologyMode, + }, + }, + want: false, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got := isHypershiftEnabled(tt.infra) + if got != tt.want { + t.Errorf("isHypershiftEnabled() = %v, want %v", got, tt.want) + } + }) + } +}🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pkg/cli/admin/upgrade/recommend/recommend_test.go` around lines 34 - 67, Refactor the TestIsAcceptRisksEnabled and TestIsHypershiftEnabled test functions to use table-driven tests instead of single-case tests. The current tests only cover positive scenarios and miss critical edge cases. For TestIsAcceptRisksEnabled, add test cases for nil featureGate input, version mismatches, disabled features, and empty feature gates. For TestIsHypershiftEnabled, add test cases for nil infra input, different control plane topology modes, and other edge conditions. Use a table structure with test case names, input parameters, and expected results, then iterate through the table to run each scenario.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@pkg/cli/admin/upgrade/recommend/alerts.go`:
- Around line 33-35: The condition checking cv == nil then attempting to
dereference cv.Status.Desired.Version will cause a nil pointer panic because cv
cannot be dereferenced when it is nil. Change the condition from cv == nil to cv
!= nil in the if statement at line 33 to ensure the ClusterVersion object is not
nil before accessing its Status.Desired.Version field in the
isAcceptRisksEnabled function call.
- Around line 27-29: The three API calls FeatureGates().Get(),
Infrastructures().Get(), and ClusterVersions().Get() are silently discarding
their error returns using the blank identifier. Instead of ignoring errors with
_, capture the error return values from each of these Get() calls and handle
them appropriately. Check each error and either log it, return it, or handle the
failure case explicitly so that network issues, RBAC denials, and other failures
are properly distinguished from successful operations rather than being silently
ignored.
---
Nitpick comments:
In `@pkg/cli/admin/upgrade/recommend/recommend_test.go`:
- Around line 34-67: Refactor the TestIsAcceptRisksEnabled and
TestIsHypershiftEnabled test functions to use table-driven tests instead of
single-case tests. The current tests only cover positive scenarios and miss
critical edge cases. For TestIsAcceptRisksEnabled, add test cases for nil
featureGate input, version mismatches, disabled features, and empty feature
gates. For TestIsHypershiftEnabled, add test cases for nil infra input,
different control plane topology modes, and other edge conditions. Use a table
structure with test case names, input parameters, and expected results, then
iterate through the table to run each scenario.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: a2529d24-e031-4cd3-ad50-33dc89d0cd7f
📒 Files selected for processing (3)
pkg/cli/admin/upgrade/recommend/alerts.gopkg/cli/admin/upgrade/recommend/recommend.gopkg/cli/admin/upgrade/recommend/recommend_test.go
nbottari9
force-pushed
the
1814-duplicate-warning
branch
2 times, most recently
from
June 15, 2026 15:29
d9a7f2a to
9cb58aa
Compare
hongkailiu
reviewed
Jun 15, 2026
nbottari9
force-pushed
the
1814-duplicate-warning
branch
from
June 16, 2026 19:06
b6b3fb2 to
05b6aaa
Compare
hongkailiu
reviewed
Jun 16, 2026
hongkailiu
left a comment
hongkailiu
left a comment
Member
There was a problem hiding this comment.
I believe it is pretty close.
Only a few of code refactoring for simplicity.
We are ready to take mock data from a cluster (HowTo is discussed over slack) and add them into examples and see if we do the output correctly.
Comment on lines
+26
to
+32
| skip, err := o.alertsEvaluatedByCVO(ctx) | ||
| if err != nil { | ||
| klog.Warningf("An error occured while determining if the CVO is evaluating alerts, so the client will check. %v", err) | ||
| } | ||
| if skip { | ||
| return nil, nil | ||
| } |
Member
There was a problem hiding this comment.
nit: check skip only if err==nil.
If err!=nil (i.e., fail to determine if CVO does it already), oc does it anyway.
Suggested change
| skip, err := o.alertsEvaluatedByCVO(ctx) | |
| if err != nil { | |
| klog.Warningf("An error occured while determining if the CVO is evaluating alerts, so the client will check. %v", err) | |
| } | |
| if skip { | |
| return nil, nil | |
| } | |
| if skip, err := o.alertsEvaluatedByCVO(ctx); err != nil { | |
| klog.Warningf("An error occured while determining if the CVO is evaluating alerts, so the client will check. %v", err) | |
| } else if skip { | |
| return nil, nil | |
| } |
| }{ | ||
| { | ||
| name: "no infrastructure", | ||
| expected: false, |
Member
There was a problem hiding this comment.
nit: no need to set up zero value explicitly. It applies to other cases too.
Suggested change
| expected: false, |
Member
|
/retitle OTA-1814: fix(alerts): Remove duplicate information in the recommend command |
openshift-ci
Bot
changed the title
|
@nbottari9: This pull request references OTA-1814 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target the "5.0.0" version, but no target version was set. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
openshift-ci-robot
added
the
jira/valid-reference
hongkailiu
reviewed
Jun 23, 2026
hongkailiu
reviewed
Jun 30, 2026
| @@ -0,0 +1,14 @@ | |||
| Failing=True: | |||
|
|
|||
| Reason: ClusterOperatorDegraded | |||
Member
There was a problem hiding this comment.
This is a bit of noise for our test. 🤷
Comment on lines
+73
to
+74
| "examples/5.0.0-cvo-handling-risks-cv.yaml": "1.2.3-not-important", | ||
| "examples/5.0.0-cvo-not-handling-risks-cv.yaml": "1.2.3-not-important", |
Member
There was a problem hiding this comment.
Suggested change
| "examples/5.0.0-cvo-handling-risks-cv.yaml": "1.2.3-not-important", | |
| "examples/5.0.0-cvo-not-handling-risks-cv.yaml": "1.2.3-not-important", | |
| "examples/5.0.0-cvo-handling-risks-cv.yaml": "5.0.0-ec.3", | |
| "examples/5.0.0-cvo-not-handling-risks-cv.yaml": "5.0.0-ec.3", |
I missed this in the previous rounds.
We will do some clean up manually after UPDATE=TRUE go test.
Then you will find out we need to handle issues that apply to this cluster but which were not included in from
hongkailiu
reviewed
Jul 14, 2026
| Reason: PodDisruptionBudgetAtLimit | ||
| Message: The pod disruption budget is preventing further disruption to pods. https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetAtLimit.md | ||
|
|
||
| error: issues that apply to this cluster but which were not included in --accept: ConditionalUpdateRisk,Failing |
Member
There was a problem hiding this comment.
I expect this to go to the other branch.
| if err != nil { | ||
| return fmt.Errorf("failed to determine if CVO is checking alerts: %v", err) | ||
| } | ||
| fmt.Fprintf(o.Out, "Cluster update risks are being handled by the Cluster Version Operator (CVO). Please use `oc adm upgrade accept ...` to accept risks.\n") |
Member
There was a problem hiding this comment.
Perhaps error fits better. What do you think?
Suggested change
| fmt.Fprintf(o.Out, "Cluster update risks are being handled by the Cluster Version Operator (CVO). Please use `oc adm upgrade accept ...` to accept risks.\n") | |
| return fmt.Errorf("issues that apply to this cluster and are not accepted. The command `oc adm upgrade accept` can be used to accept them: %s", strings.Join(sets.List(unaccepted), ",")) |
Contributor
Author
There was a problem hiding this comment.
Not sure, personally I think nothing wrong necessarily occurred in the code, more informing the user they should use the other command.
hongkailiu
reviewed
Jul 16, 2026
| if err != nil { | ||
| return fmt.Errorf("failed to determine if CVO is checking alerts: %v", err) | ||
| } | ||
| return fmt.Errorf("There are issues that apply to this cluster and have not been accepted. Cluster update risks are being handled by the Cluster Version Operator (CVO). Please use `oc adm upgrade accept ...` to accept them: %s\n", strings.Join(sets.List(issues), ",")) |
Member
There was a problem hiding this comment.
Suggested change
| return fmt.Errorf("There are issues that apply to this cluster and have not been accepted. Cluster update risks are being handled by the Cluster Version Operator (CVO). Please use `oc adm upgrade accept ...` to accept them: %s\n", strings.Join(sets.List(issues), ",")) | |
| return fmt.Errorf("There are issues that apply to this cluster and have not been accepted. Cluster update risks are being handled by the Cluster Version Operator (CVO). Please use `oc adm upgrade accept` to accept them if the associated risks are acceptable: %s\n", strings.Join(sets.List(issues), ",")) |
nbottari9
force-pushed
the
1814-duplicate-warning
branch
from
July 16, 2026 15:20
ef53167 to
4470fbe
Compare
…recommend` command Signed-off-by: Nicholas Bottari <nbottari9@gmail.com>
nbottari9
force-pushed
the
1814-duplicate-warning
branch
from
July 16, 2026 17:46
ce17dc8 to
ef559e5
Compare
hongkailiu
reviewed
Jul 16, 2026
| Reason: TestAlert | ||
| Message: Test alert for updates. https://github.com/openshift/runbooks/tree/master/alerts?runbook=notfound | ||
|
|
||
| error: There are issues that apply to this cluster and have not been accepted. `oc adm upgrade accept` can be used to accept them: ConditionalUpdateRisk |
Member
There was a problem hiding this comment.
Why is TestAlert missing here?
We probably need a bigger effort here and it is totally fine if you want to do it with a follow-up pull.
The effort would be working on the calculation of unaccepted when CVO handles the alert risks.
Contributor
Author
There was a problem hiding this comment.
recommend.go - line 345, should be unaccepted not issues
…VO-handling case Signed-off-by: Nicholas Bottari <nbottari9@gmail.com>
Member
|
/lgtm We will fix #2279 (comment) with a follow up. |
Contributor
|
Scheduling required tests: |
Contributor
Author
|
/verified by TestExamples, @nbottari9 |
openshift-ci-robot
added
the
verified
|
@nbottari9: This PR has been marked as verified by DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
Contributor
Author
|
/verified remove |
openshift-ci-robot
removed
the
verified
|
@nbottari9: The DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
Contributor
Author
|
/verified by TestExamples @nbottari9 |
|
@nbottari9: This PR has been marked as verified by DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
openshift-ci-robot
added
the
verified
Contributor
Author
|
/test build-rpms-from-tar |
Contributor
|
@nbottari9: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
openshift-merge-bot Bot
pushed a commit
that referenced
this pull request
Jul 21, 2026
* docs(alerts): add missing inputs for alerts, add new inputs for fg and infra. add more context Signed-off-by: Nicholas Bottari <nbottari9@gmail.com> * docs(alerts): address feedback in review. Signed-off-by: Nicholas Bottari <nbottari9@gmail.com> --------- Signed-off-by: Nicholas Bottari <nbottari9@gmail.com>
openshift-merge-bot Bot
pushed a commit
that referenced
this pull request
Jul 22, 2026
…ate-warning" This reverts commit 54b7c12, reversing changes made to ae0477d. PR #2279 introduced a regression: alertsEvaluatedByCVO() returns nil from alerts() when the ClusterUpdateAcceptRisks feature gate (TechPreview-only) is enabled, breaking `oc adm upgrade recommend` output. This is blocking the 5.0 nightly payload.
openshift-merge-bot Bot
added a commit
that referenced
this pull request
Jul 22, 2026
TRT-2817: Revert "Merge pull request #2279 from nbottari9/1814-duplicate-warning"
nbottari9
added a commit
to nbottari9/oc
that referenced
this pull request
Jul 27, 2026
…2296) * docs(alerts): add missing inputs for alerts, add new inputs for fg and infra. add more context Signed-off-by: Nicholas Bottari <nbottari9@gmail.com> * docs(alerts): address feedback in review. Signed-off-by: Nicholas Bottari <nbottari9@gmail.com> --------- Signed-off-by: Nicholas Bottari <nbottari9@gmail.com>
nbottari9
pushed a commit
to nbottari9/oc
that referenced
this pull request
Jul 27, 2026
…14-duplicate-warning" This reverts commit 54b7c12, reversing changes made to ae0477d. PR openshift#2279 introduced a regression: alertsEvaluatedByCVO() returns nil from alerts() when the ClusterUpdateAcceptRisks feature gate (TechPreview-only) is enabled, breaking `oc adm upgrade recommend` output. This is blocking the 5.0 nightly payload.
nbottari9
added a commit
to nbottari9/oc
that referenced
this pull request
Jul 27, 2026
…tari9/1814-duplicate-warning"" This reverts commit b57298a.
openshift-merge-bot Bot
pushed a commit
that referenced
this pull request
Jul 29, 2026
…4-duplicate-warning""
openshift-merge-bot Bot
added a commit
that referenced
this pull request
Jul 29, 2026
Revert "TRT-2817: Revert "Merge pull request #2279 from nbottari9/1814-duplicate-warning""
hongkailiu
added a commit
to hongkailiu/oc
that referenced
this pull request
Jul 31, 2026
Follow up openshift#2279 (comment) * When CVO handles accept risks, oc gets risks from `cv.Status.ConditionalUpdateRisks` that applies to the cluster and ignores the ones from `cv.Spec.DesiredUpdate.AcceptRisks`. Moreover, error out early if `oc adm recommend --accept` is used. * The special risk `ConditionalUpdateRisk` is considered only when CVO does NOT handle accept risks.
hongkailiu
added a commit
to hongkailiu/oc
that referenced
this pull request
Jul 31, 2026
Follow up openshift#2279 (comment) * When CVO handles accept risks, oc gets risks from `cv.Status.ConditionalUpdateRisks` that might apply to the cluster and ignores the ones from `cv.Spec.DesiredUpdate.AcceptRisks`. Moreover, error out early if `oc adm recommend --accept` is used. * The special risk `ConditionalUpdateRisk` is considered only when CVO does NOT handle accept risks.
hongkailiu
added a commit
to hongkailiu/oc
that referenced
this pull request
Jul 31, 2026
Follow up openshift#2279 (comment) * When CVO handles accept risks, oc gets risks from `cv.Status.ConditionalUpdateRisks` that might apply to the cluster and ignores the ones from `cv.Spec.DesiredUpdate.AcceptRisks`. Moreover, error out early if `oc adm recommend --accept` is used. * The special risk `ConditionalUpdateRisk` is considered only when CVO does NOT handle accept risks.
hongkailiu
added a commit
to hongkailiu/oc
that referenced
this pull request
Jul 31, 2026
Follow up openshift#2279 (comment) * When CVO handles accept risks, oc gets risks from `cv.Status.ConditionalUpdateRisks` that might apply to the cluster and ignores the ones from `cv.Spec.DesiredUpdate.AcceptRisks`. Moreover, error out early if `oc adm recommend --accept` is used. * The special risk `ConditionalUpdateRisk` is considered only when CVO does NOT handle accept risks.
nbottari9
pushed a commit
to nbottari9/oc
that referenced
this pull request
Aug 5, 2026
Follow up openshift#2279 (comment) * When CVO handles accept risks, oc gets risks from `cv.Status.ConditionalUpdateRisks` that might apply to the cluster and ignores the ones from `cv.Spec.DesiredUpdate.AcceptRisks`. Moreover, error out early if `oc adm recommend --accept` is used. * The special risk `ConditionalUpdateRisk` is considered only when CVO does NOT handle accept risks.
stbenjam
added a commit
to openshift-eng/ai-helpers
that referenced
this pull request
Aug 21, 2026
…udge (replaces #660) (#701) * test(ci): add adversarial payload false-revert eval cases Revives the three evidence-heavy payload-analysis eval cases from #660 on top of current main. These are the control-arm cases capturing payloads where the Payload Agent recommended incorrect reverts, and they exercise the skill's ability to distinguish well-supported revert candidates from false attributions. - case-018: mixed true/false attribution — keep openshift/oc#2279 while rejecting the cross-tenant etcd evidence attributed to openshift/hypershift#8871. - case-019: reject openshift/api#2920 and #2923; the apparent long operator waits come from disjoint interval arithmetic in the Origin monitor test. - case-020: reject openshift/ovn-kubernetes#3298 and openshift/cloud-provider-azure#164; reconstruct the ordered GCP/Azure infrastructure chains and distinguish triggers from amplifiers, detectors, and cleanup fallout. Cases are registered in the eval README case index. docs/index.html is regenerated via `make update` to clear pre-existing plugin-metadata drift so the strict plugin lint passes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(ci): bump ci plugin to 0.0.86 for new eval cases The check-version-bump CI gate requires a version bump for any change under plugins/ci/. Bump the plugin version and re-sync marketplace.json and docs/index.html via `make update`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(ci): add point-in-time cutoff + integrity judge, trim case fields Address review feedback on the revived payload false-revert eval cases: - Drop the excessive adversarial annotation fields (required_claims, must_not_conclude, forbidden_revert_candidates, distractors, key_evidence, discriminating_signal) from cases 018/019/020. Nothing in the payload-analysis eval consumed them; the core false-positive check is already covered by expected_candidates: []. - Keep the point-in-time metadata (payload_completed_at in input, analysis_cutoff in annotations) and extend it to the existing cases 001-014, since this is the highest-value part — it lets the eval detect hindsight/cheating. - Pass the cutoff to the skill as `--as-of {payload_completed_at}` and instruct the agent to perform a strict point-in-time analysis. - Add the point_in_time_integrity LLM judge (min_mean 4.0) that scores evidence provenance against analysis_cutoff and flags post-hoc leakage (later reverts, subsequent payload outcomes, present-day PR state). Cases without a cutoff score max, so it is a no-op for any future non-point-in-time case. Deliberately left behind #660's programmatic trace-hygiene judge (heavy and brittle) and the case_constraints adversarial-field judge. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(ci): support --as-of point-in-time cutoff in payload-analysis skill Add the --as-of TIMESTAMP flag to the payload-analysis argument contract and bound every external lookup and subagent investigation to the cutoff. Under --as-of the skill reasons only from evidence that existed when the payload completed: it ignores post-cutoff reverts, comments, and payload outcomes, caps the step-registry commit window at the cutoff, and never treats a later revert (or its absence) as causal evidence. This keeps the point-in-time eval from leaking post-cutoff signal into recommendations. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(ci): unify cutoff field name, add deterministic outcome judges Unify the point-in-time cutoff under one name: annotations.yaml now carries payload_completed_at, identical to input.yaml, instead of a differently-named analysis_cutoff. The duplication itself stays because the harness only exposes annotations.yaml to judges; the dataset schema now documents that constraint. Add deterministic check judges for the outcomes that were previously only graded by LLM rubric: expected_candidates_found (every expected candidate at/above min_confidence), no_unexpected_reverts (a hard false-revert gate — no non-RPM candidate at/above the revert threshold outside the expected set), failed_job_count_matches (tolerance 1), force_accept_matches, and expected_phase_matches. All gate at min_pass_rate 1.0, and the conditional ones use the harness if: field so inapplicable cases are skipped rather than auto-scored. Judge fixes: revert_scoring_accuracy had drifted from the skill (it described a retired 130-point rubric with a "single candidate" signal); it now attaches SKILL.md as context and judges against the current rubric, with a two-sided scale where a false revert scores 1 instead of falling through anchors written for the true-positive case. point_in_time_integrity uses if: instead of instructing the model to return max score for non-cutoff cases. Case fixes: case-008 records its real phase (Rejected in every historical run) instead of ""; case-010 notes now state the verified Insights API Gateway HTTP 500 story with all four job names; cases 010 and 012 declare expected_candidates: [] explicitly. Also enable parallelism: 3 and tag MLflow runs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(ci): address CodeRabbit findings on --as-of and eval judges SKILL.md: Step 3.6 now defines a single until_timestamp (the earlier of the window end and the --as-of cutoff) passed whole to both step-registry queries, replacing the until=<until_date>T23:59:59Z templates that would mangle a mid-day cutoff or silently re-admit same-day post-cutoff commits. Step 6.3 requests createdAt and treats post-cutoff revert state as unavailable: a pre-cutoff revert that merged after the cutoff counts as still Open. Eval: expected_candidates_found now verifies expected_failing_jobs linkage (substring match, short names vs full periodic names); yaml_results_valid requires the canonical candidate type field and only demands pr_url for non-RPM candidates; the outputs schema documents type-specific candidate fields. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(ci): progressive disclosure + report template for payload-analysis Restructure the payload-analysis skill for progressive disclosure: - Drop the "Required Skills" preamble that force-loaded the payload-results-yaml and payload-autodl-json schema skills before any work began; each is now loaded via the Skill tool at its point of use (Steps 6.5 and 8), keeping early context lean. - Move the three run-once blobs out of SKILL.md into references/ loaded at the step that needs them: the Step 4 investigation-subagent prompt (investigation-subagent.md), the report content rules (report-guide.md), and the Step 9 completeness-review prompt (completeness-review.md). The scoring rubric stays in SKILL.md — it is per-run core and the eval's revert_scoring_accuracy judge attaches SKILL.md as rubric context. - Replace ~220 lines of inline HTML fragments and partial CSS with assets/report-template.html — a complete fill-in-the-blanks page (placeholders plus BEGIN/END conditional and repeatable blocks) that Step 7 copies and fills. The old prose said "follow the styling conventions of the existing report format", which every run reinterpreted; the template is now the single source of structure and styling, so reports come out consistent across runs. New design: phase hero, stat tiles, confidence meters with a Confidence column in the reverts table, status pills, and per-payload history cells (status always encoded as text or luminance alongside color). SKILL.md drops from 882 lines / 10.1k words to 581 lines / 7.6k words. Bump ci to 0.0.87; marketplace + docs synced via make update. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(ci): report template mirrors the production Payload Agent format Replace the experimental template design with the canonical production report format (structure and CSS taken verbatim from a real claude-payload-agent run), parameterized with placeholders and BEGIN/END conditional blocks: executive summary with per-job persistence, revert verdict with the Score column and itemized rationale, no-revert and force-accept variants, blocking-jobs summary with S/F history patterns, per-job collapsible details with candidates-table/candidates-none alternatives, RHCOS changes with RPM candidates and per-hop diffs, informing tests, and adversarial review. Production runs had already drifted between each other on CSS details; the template pins the format so every run renders identically. Sync references/report-guide.md and the Step 10 checklist to the same section list. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(ci): correct case-011 expected_failed_job_count to 2 The payload had two failed blocking jobs (aggregated-aws-ovn-upgrade-5.0-major and aggregated-gcp-ovn-upgrade-5.0-micro), both from the external Insights API Gateway HTTP 500 incident. Confirmed by two independent eval runs (Codex/Harbor and a local claude-opus-4-6 run); the old count of 1 was annotation error and cost both LLM judges points against a correct analysis. Notes now name the jobs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(ci): verify revert-threshold evidence and add claim audit to review Three changes aimed at the recurring false-revert signature (four production incidents, all scoring 85-95 from correlation stacks that were never evidence-checked): - Step 6.1: the self-skepticism re-verification now runs for ANY score at or above the revert threshold (>= 85), not only on cap overflow. Every historical false revert scored 85-95 — below the old trigger. - Rubric: the error-message-match +20-30 tier now requires an observed artifact linking the failing operation to the modified code (stack frame, log line, event); shared subsystem vocabulary explicitly does not qualify and scores +10. - Step 9: the completeness reviewer gains a claim-audit pass for candidates >= 85. Scores change through exactly one mechanism: striking an itemized signal by citing the artifact evidence that contradicts it, then recomputing mechanically. Speculative objections remain inadmissible — preserving the guard against the earlier failure mode where valid reverts were downgraded on flimsy doubts. Eval config: declare score_range on the three numeric judges and move skill under execution per harness deprecation. Bump ci to 0.0.88. Gitignore /eval/ and /tmp/ local harness output. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(ci): add no_answer_access cheating detector to payload-analysis eval Deterministic judge that fails a case when the agent transcript touches annotations.yaml, traverses the eval dataset directory, or echoes annotation-only vocabulary (expected_candidates, force_accept_expected, ...). Verified retroactively clean against all 17 CI case transcripts from the PR head run and every local Luna/Opus run; the harness never stages annotations into the workspace, so any hit means the agent went looking. Scans the main transcript; separate subagent transcript files are out of reach. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(ci): narrow cheating detector to path patterns only Annotation field-name patterns could false-positive when the agent legitimately reasons about its output schema; the two path patterns (annotations.yaml, evals/cases/payload-analysis) are sufficient tells. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(ci): cheating detector scans merged events incl. subagent transcripts collect.py merges subagents/*.jsonl into per-case events.json, so scanning the serialized events covers subagent tool calls and text. Raw stdout remains the fallback when no events were captured. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * revert(ci): restore payload-analysis skill to pre-audit behavior Restores SKILL.md and completeness-review.md to the exact content the passing CI run (2089641369093017600, tested 572a6f3) executed. The unvalidated behavior changes from 8094b81 (>= 85 re-verification, error-match tier tightening, Step 9 claim audit) went out with no eval evidence, and their first measured run (2089697973024854016) regressed case-007: the claim audit stamped the historical wrong answer (cluster-version-operator#1309 @ 90) as verified while the true cause (operator-framework-olm#1256) went unscored. Measurement-side changes are kept: all judges including the no_answer_access cheating detector, annotation fixes, and .gitignore guards. Future skill behavior changes need control-vs-treatment eval evidence with repeats before merging. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(ci): correct payload-analysis ground truth for case-019 and case-020 case-019 was mislabeled as a false-revert trap with no expected candidate. The aws/azure 4.22->5.0 major-upgrade jobs fail because CVO #1427 (OTA-1997) added a Deployment-manifest template field the 4.22 CVO cannot render, so it silently skips its own Deployment during upgrade. This was quick-reverted by #1431 (TRT-2842) and re-landed as #1433, confirming #1427 as the real cause. Set has_revert_candidates=true and add #1427 (@100, aws/azure-ovn-upgrade) as the expected candidate; the api#2920/#2923 attributions remain red herrings. case-020 met all three Step 6.4 force-accept criteria — both blocking failures are temporary infrastructure (Azure 429 throttling / OSProvisioningTimedOut, GCP transient VIP reachability loss), no more than 2 blocking jobs, and 48.4h since the last accepted payload (>= 18h). Set force_accept_expected=true. Also normalize expected_failing_jobs across cases to semantic-core tokens for bidirectional substring matching, make the expected_candidates_found matcher match when either the annotation token or the reported job name contains the other, and bump the judge model to claude-opus-4-8. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(ci): raise payload-analysis LLM thresholds to 4.5 Tighten the quality bars now that the corrected ground truth produces consistently high scores: analysis_quality min_mean 3.5 -> 4.5 and revert_scoring_accuracy min_mean 3.0 -> 4.5. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(ci): make required_skill_invocations judge harness-neutral Detect skill usage via either signal so the one payload-analysis eval config scores both Claude Code and Codex runs: - Signal A (Claude Code): a normalized `Skill` tool invocation, read from the merged event stream with a raw stdout JSONL fallback. - Signal B (Codex and any harness without a Skill tool): the skill's canonical SKILL.md H1 heading appears in the transcript, meaning the skill body was loaded and run. Codex has no `Skill` tool, so the prior stdout-JSONL parse always failed there. Runner selection stays CLI-overridable (--agent codex --effort xhigh --model ...) over the claude-code defaults; no second config. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(ci): add no_model_refusal deterministic judge to payload-analysis Detect API-level model refusals (the model's safeguards flagged the request) in the agent or any subagent, and fail the case explicitly so the root cause is visible instead of surfacing downstream as a confusing "missing output files" failure. This is an environment/model signal, not a skill-quality one. Detection is structural to avoid false positives: - system record whose subtype starts with model_refusal (survives in both raw stdout stream-json and the flat merged events schema); - a synthetic assistant turn (model "<synthetic>") paired with stop_reason "refusal" on the same record — which excludes the <synthetic> 429 rate-limit record (stop_reason "stop_sequence"); - "safeguards flagged" text as corroboration only, never a standalone trigger. Reuses the no_answer_access corpus idiom (json.dumps(events) with a stdout fallback) and reports which corpus was scanned. Claude Code only for now: when the capture is not a Claude Code run (no system/init signature) the judge abstains (no score) rather than emit a misleading pass — Codex refusal capture shape is a TODO. Gated at min_pass_rate 1.0. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(ci): resolve payload-analysis dataset.path from config dir The harness resolves dataset.path against the config file's directory (EvalConfig.resolve_path uses config_dir = plugins/ci/evals), so the repo-root-relative value doubled to plugins/ci/evals/plugins/ci/evals/cases/payload-analysis and did not exist — workspace setup would error and score.py would load zero annotations. Use the config-dir-relative form (cases/payload-analysis), matching the majority of eval configs in the repo. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: regenerate docs/index.html after upstream merge Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(ci): pass expected_candidates_found on explicit no-revert cases The judge gated on `if: annotations.get('expected_candidates')`, but an explicit empty list — the documented way to declare "no revert expected" — is falsy in Python, so the judge silently skipped (n/a) instead of passing on all five no-revert cases (009, 010, 011, 012, 020). Its check body already returns a pass for an empty expectation, so gate on presence (`is not None`) like failed_job_count_matches does. The false-positive direction remains enforced by no_unexpected_reverts. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(ci): allow Codex network egress in payload-analysis eval The default workspace-write sandbox blocks DNS/network entirely, starving the snapshot-backed analysis of the live GCS/Prow, GitHub, and Sippy lookups it still needs. Enable network_access on the write sandbox via runner.settings, which only the Codex runner reads — the claude-code runner ignores it, so its behavior is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(ci): keep codex scratch off /tmp; score implicit no-revert cases Two payload-analysis eval changes: - Add a system-prompt line telling the agent (and, via relay, its subagents) to use $TMPDIR for temporary files and downloaded artifacts instead of /tmp, so codex runs don't fill the RAM-backed tmpfs. The top-level agent honors this; subagent propagation is best-effort since codex child processes don't inherit runner.system_prompt. - Broaden expected_candidates_found so a case declaring no-revert implicitly (has_revert_candidates: false with expected_candidates omitted) is scored vacuously, matching the explicit [] form. Skipped only when a case defines neither expectation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Stephen Benjamin <stbenjam+ai@redhat.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
If the
ClusterUpdateAcceptRisksfeature gate is enabled on the cluster, the CVO will include alerts in the conditional update risks. The client also checks for alerts, which can cause warnings or info messages to be printed twice.Description
We can detect if the server (CVO) is already including alerts by checking if
Recommended=Falsein theClusterVersionconditional updates. If this is set, we know that the feature gate is enabled, therefore the server is checking alerts and we don't need to on the client.Changes
ifstatement before callingprecheck()to check if the server is including alertsshouldSkipClientAlertChecking()- loops over theConditionalUpdatesand checks to see if any of them haveRecommended=False. If so, return true to indicate the server IS checking for alerts and we don't need to on the client.TestServerSideAlertRiskstest suite - Added new tests to test new functionalitySummary by CodeRabbit
Bug Fixes
Tests