OTA-861: Set Upgradeable=False when there is an upgrade in progress by hongkailiu · Pull Request #1080 · openshift/cluster-version-operator

hongkailiu · 2024-08-29T00:21:49Z

This PR add a Upgradeable which fails on Processing=True in clusterversion.status.conditions.
In other words, Upgradeable=False if an upgrade is in progress, including both minor level and patch level.

In addition, this PR syncs the upgradeable at the shutdown time to ensure the unsaved (due to by the throttle) upgradeable to be saved. This part could be a separate PR too.

For example, it blocks the upgrade to 4.16.1 until the ongoing upgrade
4.14.35 -> 4.15.29 completes.

It also covers the case 4.14.15-> 4.14.35 -> 4.15.29
where the upgrade 4.14.35 -> 4.15.29 is blocked until the upgrade
4.14.15-> 4.14.35 completes.

Note that we still allow for upgrade to 4.y+1.z''
in the middle of upgrade 4.y.z -> 4.y+1.z', even though direct upgrade
4.y.z -> 4.y+1.z'' might not be supported.
This is because the ugprade 4.y.z -> 4.y+1.z' might not be completed
up to a bug in 4.y+1.z' that has a fix in 4.(y+1).z''.
We need the retarget to it to land 4.y+1 on the cluster.

For OTA-861, the guard on retargeting to a minor level upgrade will be added with a follow up PR.

Update:
With the throttle of "upgradeable" disabled on the shutdown, the racing window between "CVO starts rolling the new version out" and "CVO gets shutdown" is very short (less than one second in the test).
We do not need to add the second guard back that was dropped in the 362e9ca.

The acceptedRisks for Y-then-Z upgrade in the dropped commit will be done with another PR.

hongkailiu · 2024-08-29T00:22:12Z

This PR is replacing #1079

hongkailiu · 2024-08-29T02:16:31Z

/test unit

DavidHurta · 2024-08-29T11:59:38Z

/cc

wking · 2024-08-29T19:18:35Z

Exercise a 4.17 -> this-pull update, so we can see Upgradeable=False while we're mid-update.

/payload-job periodic-ci-openshift-release-master-ci-4.18-upgrade-from-stable-4.17-e2e-gcp-ovn-upgrade

openshift-ci · 2024-08-29T19:18:43Z

@wking: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-release-master-ci-4.18-upgrade-from-stable-4.17-e2e-gcp-ovn-upgrade

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/7d5ec080-663b-11ef-899f-2883517bb747-0

petr-muller · 2024-09-04T11:09:18Z

/uncc

David and Trevor are involved in this one, seems enough ;)

DavidHurta · 2024-09-04T16:02:13Z

/title OTA-861: inhibit the 2nd minor version upgrade

DavidHurta · 2024-09-04T16:03:55Z

/retitle OTA-861: inhibit the 2nd minor version upgrade

openshift-ci-robot · 2024-09-04T16:04:03Z

@hongkailiu: This pull request references OTA-861 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.18.0" version, but no target version was set.

Details

In response to this:

This PR prevents the cluster to be upgraded to x.y+2.z1 while
the upgrade to x.y+1.z2 from x.y.z3 is still in progress.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

petr-muller · 2024-09-25T11:17:28Z

/test all

This commits entends the guard on the 2nd Y-stream upgrade, i.e., blocking an Y-stream upgrade if there is already an Y-stream upgrade in progress, to the guard on any Y-stream upgrade if there is already an upgrade in progress, reguardless of Y-stream or Z-stream. For example, it covers the case 4.14.15-> 4.14.35 -> 4.15.29 where the upgrade 4.14.35 -> 4.15.29 is blocked until the upgrade 4.14.15-> 4.14.35 completes. Note that we still allow for upgrade to 4.y+1.z'' in the middle of upgrade 4.y.z -> 4.y+1.z', even though direct upgrade 4.y.z -> 4.y+1.z'' might not be supported. This is because the ugprade 4.y.z -> 4.y+1.z' might not be completed up to a bug in 4.y+1.z' that has a fix in 4.(y+1).z''. We need the retarget to it to land 4.y+1 on the cluster.

openshift-ci-robot · 2024-10-01T04:44:38Z

@hongkailiu: This pull request references OTA-861 which is a valid jira issue.

Details

In response to this:

This PR prevents the cluster to be upgraded to x.y+2.z1 while
the upgrade to x.y+1.z2 from x.y.z3 is still in progress.

Update:
Block Y stream upgrade if any upgrade is in progress
This commits entends the guard on the 2nd Y-stream upgrade,
i.e., blocking an Y-stream upgrade if there is already an
Y-stream upgrade in progress,
to the guard on any Y-stream upgrade if there is already an
upgrade in progress, reguardless of Y-stream or Z-stream.

For example, it covers the case 4.14.15-> 4.14.35 -> 4.15.29
where the upgrade 4.14.35 -> 4.15.29 is blocked until the upgrade
4.14.15-> 4.14.35 completes.

Note that we still allow for upgrade to 4.y+1.z''
in the middle of upgrade 4.y.z -> 4.y+1.z', even though direct upgrade
4.y.z -> 4.y+1.z'' might not be supported.
This is because the ugprade 4.y.z -> 4.y+1.z' might not be completed
up to a bug in 4.y+1.z' that has a fix in 4.(y+1).z''.
We need the retarget to it to land 4.y+1 on the cluster.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2024-10-01T14:27:36Z

@hongkailiu: This pull request references OTA-861 which is a valid jira issue.

Details

In response to this:

This PR adds a guard that blocks an Y-stream upgrade
if there is already an upgrade in progress, reguardless of Y-stream or Z-stream.

For example, it blocks the upgrade to 4.16.1 until the ongoing upgrade
4.14.35 -> 4.15.29 completes.

It also covers the case 4.14.15-> 4.14.35 -> 4.15.29
where the upgrade 4.14.35 -> 4.15.29 is blocked until the upgrade
4.14.15-> 4.14.35 completes.

Note that we still allow for upgrade to 4.y+1.z''
in the middle of upgrade 4.y.z -> 4.y+1.z', even though direct upgrade
4.y.z -> 4.y+1.z'' might not be supported.
This is because the ugprade 4.y.z -> 4.y+1.z' might not be completed
up to a bug in 4.y+1.z' that has a fix in 4.(y+1).z''.
We need the retarget to it to land 4.y+1 on the cluster.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

This commit generates a message in the accepted risks for the unblocking case 4.y.z -> 4.y+1.z' -> 4.y+1.z''.

petr-muller · 2024-10-02T18:17:47Z

openshift/release#57408 should fix the hypershift jobs

openshift-ci-robot · 2024-10-04T18:34:39Z

@hongkailiu: This pull request references OTA-861 which is a valid jira issue.

Details

In response to this:

This PR add a Upgradeable which fails on Processing=True in clusterversion.status.conditions.
In other words, Upgradeable=False if an upgrade is in progress, including both minor level and patch level.

In addition, this PR syncs the upgradeable at the shutdown time to ensure the unsaved (due to by the throttle) upgradeable to be saved. This part could be a separate PR too.

This PR adds a guard that blocks an Y-stream upgrade
if there is already an upgrade in progress, reguardless of Y-stream or Z-stream.

For example, it blocks the upgrade to 4.16.1 until the ongoing upgrade
4.14.35 -> 4.15.29 completes.

It also covers the case 4.14.15-> 4.14.35 -> 4.15.29
where the upgrade 4.14.35 -> 4.15.29 is blocked until the upgrade
4.14.15-> 4.14.35 completes.

Note that we still allow for upgrade to 4.y+1.z''
in the middle of upgrade 4.y.z -> 4.y+1.z', even though direct upgrade
4.y.z -> 4.y+1.z'' might not be supported.
This is because the ugprade 4.y.z -> 4.y+1.z' might not be completed
up to a bug in 4.y+1.z' that has a fix in 4.(y+1).z''.
We need the retarget to it to land 4.y+1 on the cluster.

For OTA-861, the guard on retargeting to a minor level upgrade will be added with a follow up PR.

Update:
With the throttle of "upgradeable" disabled on the shutdown, the racing window between "CVO starts rolling the new version out" and "CVO gets shutdown" is very short (less than one second in the test).
We do not need to add the second guard back that was dropped in the 362e9ca.

The acceptedRisks for Y-then-Z upgrade in the dropped commit will be done with another PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

hongkailiu · 2024-10-04T19:30:05Z

/test unit

wking

0366e77 looks good to me too.

/lgtm

openshift-ci-robot · 2024-10-04T21:25:53Z

@hongkailiu: This pull request references OTA-861 which is a valid jira issue.

Details

In response to this:

This PR add a Upgradeable which fails on Processing=True in clusterversion.status.conditions.
In other words, Upgradeable=False if an upgrade is in progress, including both minor level and patch level.

In addition, this PR syncs the upgradeable at the shutdown time to ensure the unsaved (due to by the throttle) upgradeable to be saved. This part could be a separate PR too.

For example, it blocks the upgrade to 4.16.1 until the ongoing upgrade
4.14.35 -> 4.15.29 completes.

It also covers the case 4.14.15-> 4.14.35 -> 4.15.29
where the upgrade 4.14.35 -> 4.15.29 is blocked until the upgrade
4.14.15-> 4.14.35 completes.

Note that we still allow for upgrade to 4.y+1.z''
in the middle of upgrade 4.y.z -> 4.y+1.z', even though direct upgrade
4.y.z -> 4.y+1.z'' might not be supported.
This is because the ugprade 4.y.z -> 4.y+1.z' might not be completed
up to a bug in 4.y+1.z' that has a fix in 4.(y+1).z''.
We need the retarget to it to land 4.y+1 on the cluster.

For OTA-861, the guard on retargeting to a minor level upgrade will be added with a follow up PR.

Update:
With the throttle of "upgradeable" disabled on the shutdown, the racing window between "CVO starts rolling the new version out" and "CVO gets shutdown" is very short (less than one second in the test).
We do not need to add the second guard back that was dropped in the 362e9ca.

The acceptedRisks for Y-then-Z upgrade in the dropped commit will be done with another PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

hongkailiu · 2024-10-06T23:22:48Z

/test e2e-hypershift

hongkailiu · 2024-10-08T16:21:27Z

/test unit

wking

/lgtm

openshift-ci · 2024-11-08T13:08:04Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hongkailiu, petr-muller, wking

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [petr-muller,wking]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

petr-muller · 2024-11-08T14:29:51Z

/retest

evakhoni · 2024-11-24T16:30:57Z

pre-merge verified successfully in all four varients
/label qe-approved

openshift-ci-robot · 2024-11-24T16:31:01Z

@hongkailiu: This pull request references OTA-861 which is a valid jira issue.

Details

In response to this:

This PR add a Upgradeable which fails on Processing=True in clusterversion.status.conditions.
In other words, Upgradeable=False if an upgrade is in progress, including both minor level and patch level.

In addition, this PR syncs the upgradeable at the shutdown time to ensure the unsaved (due to by the throttle) upgradeable to be saved. This part could be a separate PR too.

For example, it blocks the upgrade to 4.16.1 until the ongoing upgrade
4.14.35 -> 4.15.29 completes.

It also covers the case 4.14.15-> 4.14.35 -> 4.15.29
where the upgrade 4.14.35 -> 4.15.29 is blocked until the upgrade
4.14.15-> 4.14.35 completes.

Note that we still allow for upgrade to 4.y+1.z''
in the middle of upgrade 4.y.z -> 4.y+1.z', even though direct upgrade
4.y.z -> 4.y+1.z'' might not be supported.
This is because the ugprade 4.y.z -> 4.y+1.z' might not be completed
up to a bug in 4.y+1.z' that has a fix in 4.(y+1).z''.
We need the retarget to it to land 4.y+1 on the cluster.

For OTA-861, the guard on retargeting to a minor level upgrade will be added with a follow up PR.

Update:
With the throttle of "upgradeable" disabled on the shutdown, the racing window between "CVO starts rolling the new version out" and "CVO gets shutdown" is very short (less than one second in the test).
We do not need to add the second guard back that was dropped in the 362e9ca.

The acceptedRisks for Y-then-Z upgrade in the dropped commit will be done with another PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

wking · 2024-11-25T23:47:24Z

operators should not create watch channels very often is unrelated:

/override ci/prow/e2e-agnostic-ovn

openshift-ci · 2024-11-25T23:48:40Z

@wking: Overrode contexts on behalf of wking: ci/prow/e2e-agnostic-ovn

Details

In response to this:

operators should not create watch channels very often is unrelated:

/override ci/prow/e2e-agnostic-ovn

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-ci · 2024-11-25T23:48:43Z

@hongkailiu: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-bot · 2024-11-26T05:09:25Z

[ART PR BUILD NOTIFIER]

Distgit: cluster-version-operator
This PR has been included in build cluster-version-operator-container-v4.19.0-202411260335.p0.gb6b7345.assembly.stream.el9.
All builds following this will include this PR.

openshift-ci Bot requested review from petr-muller and wking August 29, 2024 00:22

hongkailiu mentioned this pull request Aug 29, 2024

OTA-861: inhibit the 2nd minor version upgrade #1079

Closed

hongkailiu force-pushed the OTA-861-better branch from 0c7bbfe to 1d83493 Compare August 29, 2024 01:32

openshift-ci Bot requested a review from DavidHurta August 29, 2024 11:59

inhibit the 2nd minor version upgrade

741db89

hongkailiu force-pushed the OTA-861-better branch from 1d83493 to 741db89 Compare August 29, 2024 18:23

openshift-ci Bot removed the request for review from petr-muller September 4, 2024 11:09

openshift-ci Bot changed the title ~~inhibit the 2nd minor version upgrade~~ OTA-861: inhibit the 2nd minor version upgrade Sep 4, 2024

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Sep 4, 2024

hongkailiu force-pushed the OTA-861-better branch from 3412026 to 5ea5ebe Compare October 1, 2024 04:41

hongkailiu changed the title ~~OTA-861: inhibit the 2nd minor version upgrade~~ OTA-861: Block Y stream upgrade if any upgrade is in progress Oct 1, 2024

Generate accepted risks in an unblocking case

8aacdac

This commit generates a message in the accepted risks for the unblocking case 4.y.z -> 4.y+1.z' -> 4.y+1.z''.

wking reviewed Oct 1, 2024

View reviewed changes

Comment thread pkg/cvo/upgradeable.go Outdated

wking reviewed Oct 2, 2024

View reviewed changes

Comment thread pkg/payload/precondition/clusterversion/upgradeable.go Outdated

wking reviewed Oct 2, 2024

View reviewed changes

Comment thread pkg/payload/precondition/clusterversion/upgradeable_test.go

openshift-ci Bot removed the lgtm Indicates that a PR is ready to be merged. label Oct 4, 2024

wking approved these changes Oct 4, 2024

View reviewed changes

openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Oct 4, 2024

petr-muller approved these changes Oct 7, 2024

View reviewed changes

openshift-ci Bot assigned petr-muller Oct 7, 2024

openshift-ci Bot removed the lgtm Indicates that a PR is ready to be merged. label Oct 8, 2024

hongkailiu force-pushed the OTA-861-better branch from 9e6149b to 0366e77 Compare October 9, 2024 10:47

wking approved these changes Nov 4, 2024

View reviewed changes

openshift-ci Bot added lgtm Indicates that a PR is ready to be merged. and removed lgtm Indicates that a PR is ready to be merged. labels Nov 4, 2024

Improve condition's reason and message

7ad9622

hongkailiu force-pushed the OTA-861-better branch from c1f693a to 7ad9622 Compare November 5, 2024 15:15

petr-muller approved these changes Nov 8, 2024

View reviewed changes

openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Nov 8, 2024

openshift-ci Bot added the qe-approved Signifies that QE has signed off on this PR label Nov 24, 2024

evakhoni mentioned this pull request Nov 25, 2024

Automate OCP-25473 openshift/release#58136

Merged

openshift-merge-bot Bot merged commit b6b7345 into openshift:master Nov 25, 2024

Uh oh!

Conversation

hongkailiu commented Aug 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hongkailiu commented Aug 29, 2024

Uh oh!

hongkailiu commented Aug 29, 2024

Uh oh!

DavidHurta commented Aug 29, 2024

Uh oh!

wking commented Aug 29, 2024

Uh oh!

openshift-ci Bot commented Aug 29, 2024

Uh oh!

petr-muller commented Sep 4, 2024

Uh oh!

DavidHurta commented Sep 4, 2024 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DavidHurta commented Sep 4, 2024 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Sep 4, 2024 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

petr-muller commented Sep 25, 2024

Uh oh!

openshift-ci-robot commented Oct 1, 2024 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Oct 1, 2024 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

petr-muller commented Oct 2, 2024

Uh oh!

openshift-ci-robot commented Oct 4, 2024 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hongkailiu commented Oct 4, 2024

Uh oh!

wking left a comment

Choose a reason for hiding this comment

Uh oh!

openshift-ci-robot commented Oct 4, 2024 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hongkailiu commented Oct 6, 2024

Uh oh!

hongkailiu commented Oct 8, 2024

Uh oh!

wking left a comment

Choose a reason for hiding this comment

Uh oh!

openshift-ci Bot commented Nov 8, 2024

Uh oh!

petr-muller commented Nov 8, 2024

Uh oh!

evakhoni commented Nov 24, 2024

Uh oh!

openshift-ci-robot commented Nov 24, 2024 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wking commented Nov 25, 2024

Uh oh!

openshift-ci Bot commented Nov 25, 2024

Uh oh!

openshift-ci Bot commented Nov 25, 2024

Uh oh!

openshift-bot commented Nov 26, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

hongkailiu commented Aug 29, 2024 •

edited

Loading

DavidHurta commented Sep 4, 2024 •

edited by openshift-ci Bot

Loading

DavidHurta commented Sep 4, 2024 •

edited by openshift-ci Bot

Loading

openshift-ci-robot commented Sep 4, 2024 •

edited by openshift-ci Bot

Loading

openshift-ci-robot commented Oct 1, 2024 •

edited by openshift-ci Bot

Loading

openshift-ci-robot commented Oct 1, 2024 •

edited by openshift-ci Bot

Loading

openshift-ci-robot commented Oct 4, 2024 •

edited by openshift-ci Bot

Loading

openshift-ci-robot commented Oct 4, 2024 •

edited by openshift-ci Bot

Loading

openshift-ci-robot commented Nov 24, 2024 •

edited by openshift-ci Bot

Loading