OCPBUGS-42880: Upgradeable=False should not block a 4.(y+1).z to 4.(y+1).z' retarget by hongkailiu · Pull Request #1094 · openshift/cluster-version-operator

hongkailiu · 2024-10-09T01:37:39Z

https://issues.redhat.com/browse/OCPBUGS-42880

openshift-ci-robot · 2024-10-09T01:37:44Z

@hongkailiu: This pull request references Jira Issue OCPBUGS-42880, which is invalid:

expected the bug to target the "4.18.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

https://issues.redhat.com/browse/OCPBUGS-42880

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

hongkailiu · 2024-10-09T02:16:39Z

/jira refresh

openshift-ci-robot · 2024-10-09T02:16:45Z

@hongkailiu: This pull request references Jira Issue OCPBUGS-42880, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.18.0) matches configured target version for branch (4.18.0)
bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @jiajliu

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

evakhoni · 2024-10-09T10:40:17Z

/cc

hongkailiu · 2024-10-09T13:36:58Z

/retest-required

hongkailiu · 2024-10-09T20:34:26Z

/retest-required

hongkailiu · 2024-10-10T18:31:08Z

/test e2e-aws-ovn-techpreview

hongkailiu · 2024-10-11T19:31:02Z

confirming a reproduction with 4.17.0 -> 4.18.0-ec.1 -> 4.18.0-ec.2.

### cluster bot: launch 4.17.0 aws

$ oc get clusterversion version
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.17.0    True        False         4m28s   Cluster version is 4.17.0

$ oc adm upgrade channel candidate-4.18
$ oc adm upgrade --to 4.18.0-ec.1

### Wait until the update has been accepted ...

oc adm upgrade
info: An upgrade is in progress. Working towards 4.18.0-ec.1: 110 of 901 done (12% complete), waiting on etcd, kube-apiserver
...

$ oc -n openshift-config-managed patch configmap admin-gates --type json -p '[ {"op": "add", "path": "/data/ack-4.17-testing", value: "testing"}]'

$ oc  get -n openshift-config-managed configmap admin-gates -o yaml | yq .data -y
ack-4.17-testing: testing

$ oc adm upgrade
info: An upgrade is in progress. Working towards 4.18.0-ec.1: 110 of 901 done (12% complete), waiting on etcd, kube-apiserver

Upgradeable=False

  Reason: AdminAckRequired
  Message: testing

$ oc adm upgrade --to 4.18.0-ec.2 --allow-upgrade-with-warnings

### it is blocked
$ oc adm upgrade
info: An upgrade is in progress. Working towards 4.18.0-ec.1: 110 of 901 done (12% complete), waiting on etcd, kube-apiserver

Upgradeable=False

  Reason: AdminAckRequired
  Message: testing

ReleaseAccepted=False

  Reason: PreconditionChecks
  Message: Preconditions failed for payload loaded version="4.18.0-ec.2" image="quay.io/openshift-release-dev/ocp-release@sha256:a3233d18821af30610bc16d4a3d0f45a5f6bcfbb50eda139b8b82b824a78559f": Precondition "ClusterVersionUpgradeable" failed because of "AdminAckRequired": testing

Upstream: https://api.integration.openshift.com/api/upgrades_info/graph
Channel: candidate-4.18 (available channels: candidate-4.18)

Recommended updates:

  VERSION     IMAGE
  4.18.0-ec.2 quay.io/openshift-release-dev/ocp-release@sha256:a3233d18821af30610bc16d4a3d0f45a5f6bcfbb50eda139b8b82b824a78559f

wking · 2024-10-11T20:32:18Z

With a new 4.17.1 cluster (logs), poking at the rollback user experience:

$ oc adm upgrade --allow-explicit-upgrade --force --to-image registry.build09.ci.openshift.org/ci-ln-jpktmvt/release@sha256:bb98538f1fc1fd9deee7efe3397a6a70ddee80b298eb6c0077795fd3eb63cf34
warning: The requested upgrade image is not one of the available updates. You have used --allow-explicit-upgrade for the update to proceed anyway
warning: --force overrides cluster verification of your supplied release image and waives any update precondition failures.
Requested update to release image registry.build09.ci.openshift.org/ci-ln-jpktmvt/release@sha256:bb98538f1fc1fd9deee7efe3397a6a70ddee80b298eb6c0077795fd3eb63cf34

And once that's going:

$ oc -n openshift-config-managed patch configmap admin-gates --type json -p '[{"op": "add", "path": "/data", "value": {"ack-4.17-testing": "testing"}}]'

And once that's going, ask to go back to 4.17.1:

$ registry.build09.ci.openshift.org/ci-ln-38qy8yk/release@sha256:e16ac60ac6971e5b6f89c1d818f5ae711c0d63ad6a6a26ffe795c738e8cc4dde
$ oc adm upgrade --allow-explicit-upgrade --allow-upgrade-with-warnings --to-image registry.build09.ci.openshift.org/ci-ln-38qy8yk/release@sha256:e16ac60ac6971e5b6f89c1d818f5ae711c0d63ad6a6a26f
fe795c738e8cc4dde
warning: The requested upgrade image is not one of the available updates. You have used --allow-explicit-upgrade for the update to proceed anyway
warning: --allow-upgrade-with-warnings is bypassing: the cluster is already upgrading:

  Reason: ClusterOperatorsUpdating
  Message: Working towards 4.18.0-0.test-2024-10-11-181029-ci-ln-jpktmvt-latest: 110 of 890 done (12% complete), waiting on etcd, kube-apiserver
Requested update to release image registry.build09.ci.openshift.org/ci-ln-38qy8yk/release@sha256:e16ac60ac6971e5b6f89c1d818f5ae711c0d63ad6a6a26ffe795c738e8cc4dde

Which the Rollback precondition rejects, but, new with this pull, the Upgrade condition is ok with:

$ oc get -o json clusterversion version | jq -r '.status.conditions[] | select(.type == "ReleaseAccepted").message'
Preconditions failed for payload loaded version="4.17.1" image="registry.build09.ci.openshift.org/ci-ln-38qy8yk/release@sha256:e16ac60ac6971e5b6f89c1d818f5ae711c0d63ad6a6a26ffe795c738e8cc4dde": Multiple precondition checks failed:
* Precondition "ClusterVersionRollback" failed because of "LowDesiredVersion": 4.17.1 is less than the current target 4.18.0-0.test-2024-10-11-181029-ci-ln-jpktmvt-latest and matches the cluster's previous version, but rollbacks that change major or minor versions are not recommended
* Precondition "ClusterVersionRecommendedUpdate" failed because of "UnknownUpdate": RetrievedUpdates=False (VersionNotFound), so the recommended status of updating from 4.18.0-0.test-2024-10-11-181029-ci-ln-jpktmvt-latest to 4.17.1 is unknown.

Give up on that rollback request:

$ oc adm upgrade --clear
Cancelled requested upgrade to registry.build09.ci.openshift.org/ci-ln-38qy8yk/release@sha256:e16ac60ac6971e5b6f89c1d818f5ae711c0d63ad6a6a26ffe795c738e8cc4dde

Ask to downgrade to an earlier 4.18:

$ oc adm release info registry.ci.openshift.org/ocp/release:4.18.0-0.nightly-2024-10-10-195326 | head -n3
Name:           4.18.0-0.nightly-2024-10-10-195326
Digest:         sha256:dbbaaf5235c90c4aafb24292da4d4e22685108bd01f70005703bd17576489d86
Created:        2024-10-10T19:55:13Z
$ oc adm upgrade --allow-explicit-upgrade --allow-upgrade-with-warnings --to-image registry.ci.openshift.org/ocp/release@sha256:dbbaaf5235c90c4aafb24292da4d4e22685108bd01f70005703bd17576489d86
warning: The requested upgrade image is not one of the available updates. You have used --allow-explicit-upgrade for the update to proceed anyway
warning: --allow-upgrade-with-warnings is bypassing: the cluster is already upgrading:

  Reason: ClusterOperatorsUpdating
  Message: Working towards 4.18.0-0.test-2024-10-11-181029-ci-ln-jpktmvt-latest: 726 of 890 done (81% complete), waiting on dns, network
Requested update to release image registry.ci.openshift.org/ocp/release@sha256:dbbaaf5235c90c4aafb24292da4d4e22685108bd01f70005703bd17576489d86

And it's also rejected, but on the earlier signature-check, so we will need to wait until we have an Engineering Candidate with this pull request in it before we can test Upgradeble not being involved in that anymore.

$ oc get -o json clusterversion version | jq -r '.status.conditions[] | select(.type == "ReleaseAccepted").message'
Retrieving payload failed version="" image="registry.ci.openshift.org/ocp/release@sha256:dbbaaf5235c90c4aafb24292da4d4e22685108bd01f70005703bd17576489d86" failure=The update cannot be verified: unable to verify sha256:dbbaaf5235c90c4aafb24292da4d4e22685108bd01f70005703bd17576489d86 against keyrings: verifier-public-key-redhat

evakhoni · 2024-10-14T15:43:52Z

 				clusterVersion.Status.Conditions = append(clusterVersion.Status.Conditions, configv1.ClusterOperatorStatusCondition{
 					Type:    configv1.OperatorUpgradeable,
 					Status:  *tc.upgradeable,
+					Reason:  "bla",


wking · 2024-10-15T05:54:33Z

/cherrypick release-4.17

openshift-cherrypick-robot · 2024-10-15T05:54:36Z

@wking: once the present PR merges, I will cherry-pick it on top of release-4.17 in a new PR and assign it to you.

Details

In response to this:

/cherrypick release-4.17

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

petr-muller

LGTM with a nit, feel free to unhold

/hold

Co-authored-by: Petr Muller <afri@afri.cz>

hongkailiu · 2024-10-15T13:01:14Z

/label tide/merge-method-squash

petr-muller

Thanks!

/hold cancel

openshift-ci · 2024-10-15T13:17:39Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hongkailiu, petr-muller, wking

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [petr-muller,wking]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci · 2024-10-15T15:57:55Z

@hongkailiu: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

evakhoni · 2024-10-16T20:23:45Z

pre-merge verified in https://issues.redhat.com/browse/OCPBUGS-42880?focusedId=25803260&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-25803260
/label qe-approved
/hold
I have mentioned 3 minor issues observed during the verification. please review, and unhold if not planning to address within the current PR. thanks!

openshift-ci-robot · 2024-10-16T20:23:53Z

@hongkailiu: This pull request references Jira Issue OCPBUGS-42880, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.18.0) matches configured target version for branch (4.18.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @evakhoni

Details

In response to this:

https://issues.redhat.com/browse/OCPBUGS-42880

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

evakhoni · 2024-10-20T18:24:10Z

following the findings and retest in OCPBUGS-42880#comment-25883694 the issues are not directly related to the fix. passing as verified.
/unhold

openshift-ci-robot · 2024-10-20T18:26:32Z

@hongkailiu: Jira Issue OCPBUGS-42880: All pull requests linked via external trackers have merged:

openshift/cluster-version-operator#1094

Jira Issue OCPBUGS-42880 has been moved to the MODIFIED state.

Details

In response to this:

https://issues.redhat.com/browse/OCPBUGS-42880

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-cherrypick-robot · 2024-10-20T18:27:17Z

@wking: new pull request created: #1095

Details

In response to this:

/cherrypick release-4.17

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-bot · 2024-10-20T22:44:23Z

[ART PR BUILD NOTIFIER]

Distgit: cluster-version-operator
This PR has been included in build cluster-version-operator-container-v4.18.0-202410202210.p0.gadf0cf5.assembly.stream.el9.
All builds following this will include this PR.

openshift-ci Bot requested review from petr-muller and wking October 9, 2024 01:38

hongkailiu force-pushed the OCPBUGS-42880 branch 3 times, most recently from 0cdab63 to fce734d Compare October 9, 2024 02:01

openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Oct 9, 2024

openshift-ci Bot requested a review from jiajliu October 9, 2024 02:16

wking reviewed Oct 9, 2024

View reviewed changes

Comment thread pkg/cvo/upgradeable.go Outdated

hongkailiu force-pushed the OCPBUGS-42880 branch 2 times, most recently from 9eb95d3 to efc4c61 Compare October 9, 2024 10:33

openshift-ci Bot requested a review from evakhoni October 9, 2024 10:40

wking reviewed Oct 10, 2024

View reviewed changes

Comment thread pkg/payload/precondition/clusterversion/upgradeable.go Outdated

wking reviewed Oct 10, 2024

View reviewed changes

Comment thread pkg/payload/precondition/clusterversion/upgradeable.go Outdated

wking reviewed Oct 10, 2024

View reviewed changes

Comment thread pkg/payload/precondition/clusterversion/upgradeable_test.go Outdated

wking reviewed Oct 10, 2024

View reviewed changes

Comment thread pkg/payload/precondition/clusterversion/upgradeable_test.go Outdated

hongkailiu force-pushed the OCPBUGS-42880 branch 3 times, most recently from 73aa5f9 to 8cf5d7a Compare October 11, 2024 01:23

wking reviewed Oct 11, 2024

View reviewed changes

Comment thread pkg/payload/precondition/clusterversion/upgradeable.go Outdated

evakhoni reviewed Oct 14, 2024

View reviewed changes

petr-muller approved these changes Oct 15, 2024

View reviewed changes

Comment thread pkg/payload/precondition/clusterversion/upgradeable.go Outdated

openshift-ci Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 15, 2024

openshift-ci Bot assigned petr-muller Oct 15, 2024

Update pkg/payload/precondition/clusterversion/upgradeable.go

ed8d221

Co-authored-by: Petr Muller <afri@afri.cz>

openshift-ci Bot removed the lgtm Indicates that a PR is ready to be merged. label Oct 15, 2024

openshift-ci Bot added the tide/merge-method-squash Denotes a PR that should be squashed by tide when it merges. label Oct 15, 2024

petr-muller approved these changes Oct 15, 2024

View reviewed changes

openshift-ci Bot added lgtm Indicates that a PR is ready to be merged. and removed do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. labels Oct 15, 2024

hongkailiu mentioned this pull request Oct 15, 2024

OTA-861: Generate an accepted risk for Y-then-Z upgrade #1093

Merged

openshift-ci Bot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. qe-approved Signifies that QE has signed off on this PR labels Oct 16, 2024

openshift-ci Bot requested a review from evakhoni October 16, 2024 20:23

openshift-ci Bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 20, 2024

openshift-merge-bot Bot merged commit adf0cf5 into openshift:master Oct 20, 2024

openshift-cherrypick-robot mentioned this pull request Oct 20, 2024

[release-4.17] OCPBUGS-43586: Upgradeable=False should not block a 4.(y+1).z to 4.(y+1).z' retarget #1095

Merged

Uh oh!

Conversation

hongkailiu commented Oct 9, 2024

Uh oh!

openshift-ci-robot commented Oct 9, 2024

Uh oh!

hongkailiu commented Oct 9, 2024

Uh oh!

openshift-ci-robot commented Oct 9, 2024

Uh oh!

Uh oh!

evakhoni commented Oct 9, 2024

Uh oh!

hongkailiu commented Oct 9, 2024

Uh oh!

hongkailiu commented Oct 9, 2024

Uh oh!

hongkailiu commented Oct 10, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hongkailiu commented Oct 11, 2024

Uh oh!

wking commented Oct 11, 2024

Uh oh!

evakhoni Oct 14, 2024

Choose a reason for hiding this comment

Uh oh!

wking commented Oct 15, 2024

Uh oh!

openshift-cherrypick-robot commented Oct 15, 2024

Uh oh!

petr-muller left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hongkailiu commented Oct 15, 2024

Uh oh!

petr-muller left a comment

Choose a reason for hiding this comment

Uh oh!

openshift-ci Bot commented Oct 15, 2024

Uh oh!

openshift-ci Bot commented Oct 15, 2024

Uh oh!

evakhoni commented Oct 16, 2024

Uh oh!

openshift-ci-robot commented Oct 16, 2024

Uh oh!

evakhoni commented Oct 20, 2024

Uh oh!

openshift-ci-robot commented Oct 20, 2024

Uh oh!

openshift-cherrypick-robot commented Oct 20, 2024

Uh oh!

openshift-bot commented Oct 20, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants