OCPBUGS-23126: Fix a bug on deletion of a hostedcluster by nunnatsa · Pull Request #3234 · openshift/hypershift

nunnatsa · 2023-11-26T13:49:58Z

What this PR does / why we need it

A user destroying a HostedCluster can cause the HostedCluster to hang indefinitely if the destroy command times out during execution.

This is due to the hcp cli placing a finalizer on the HostedCluster during deletion which the cli tool later removes after waiting for some clean up actions to occur. If a user cancels the hcp destroy cluster command (or the command times out) while the cli is waiting for cleanup, then the HostedCluster will hang indefinitely with a DeletionTimestamp != nil.

This PR only performs the additional cleaning logic, including the setting and removing the special finalizer, for platform that actually creates additional resources. Also, this PR sets the hostedCluster as the owner for these secrets, so they will be deleted with the HostedCluster.

Which issue(s) this PR fixes

Fixes #OCPBUGS-23126

Checklist

Subject and description added to both, commit and PR.
Relevant issues have been referenced.
This change includes docs.
This change includes unit tests.

openshift-ci-robot · 2023-11-26T13:50:03Z

@nunnatsa: This pull request references Jira Issue OCPBUGS-23126, which is invalid:

expected the bug to target the "4.15.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

What this PR does / why we need it:

Which issue(s) this PR fixes (optional, use fixes #<issue_number>(, fixes #<issue_number>, ...) format, where issue_number might be a GitHub issue, or a Jira story:
Fixes #

Checklist

Subject and description added to both, commit and PR.

Relevant issues have been referenced.

This change includes docs.

This change includes unit tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

nunnatsa · 2023-11-26T20:54:57Z

/test e2e-aws
/test e2e-kubevirt-aws-ovn

nunnatsa · 2023-11-27T07:17:00Z

/jira refresh

openshift-ci-robot · 2023-11-27T07:17:05Z

@nunnatsa: This pull request references Jira Issue OCPBUGS-23126, which is invalid:

expected the bug to target the "4.15.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

nunnatsa · 2023-11-27T07:19:27Z

/jira refresh

openshift-ci-robot · 2023-11-27T07:19:35Z

@nunnatsa: This pull request references Jira Issue OCPBUGS-23126, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.15.0) matches configured target version for branch (4.15.0)
bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @LiangquanLi930

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2023-11-28T13:05:33Z

@nunnatsa: This pull request references Jira Issue OCPBUGS-23126, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.15.0) matches configured target version for branch (4.15.0)
bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @LiangquanLi930

Details

In response to this:

What this PR does / why we need it

A user destroying a HostedCluster can cause the HostedCluster to
hang indefinitely if the destroy command times out during execution

This is due to the hcp cli placing a finalizer on the HostedCluster
during deletion which the cli tool later removes after waiting for
some clean up actions to occur. If a user cancels the hcp destroy cluster command (or the command times out) while the cli is waiting
for cleanup, then the HostedCluster will hang indefinitely with a
DeletionTimestamp != nil.

This PR only performs the additional cleaning logic, including the
setting and removing the special finalizer, for platform that
actually creates additional resources. Also, this PR removes the
secret cleaning from the cmd, and instead, set the hostedCluster
as the owner for these secrets, so they will be deleted with the
HostedCluster.

Which issue(s) this PR fixes

Fixes #OCPBUGS-23126

Checklist

Subject and description added to both, commit and PR.

Relevant issues have been referenced.

This change includes docs.

This change includes unit tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2023-11-28T13:06:26Z

@nunnatsa: This pull request references Jira Issue OCPBUGS-23126, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.15.0) matches configured target version for branch (4.15.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @LiangquanLi930

Details

In response to this:

What this PR does / why we need it

A user destroying a HostedCluster can cause the HostedCluster to hang indefinitely if the destroy command times out during execution.

This is due to the hcp cli placing a finalizer on the HostedCluster during deletion which the cli tool later removes after waiting for some clean up actions to occur. If a user cancels the hcp destroy cluster command (or the command times out) while the cli is waiting for cleanup, then the HostedCluster will hang indefinitely with a DeletionTimestamp != nil.

This PR only performs the additional cleaning logic, including the setting and removing the special finalizer, for platform that actually creates additional resources. Also, this PR removes the secret cleaning from the cmd, and instead, set the hostedCluster as the owner for these secrets, so they will be deleted with the HostedCluster.

Which issue(s) this PR fixes

Fixes #OCPBUGS-23126

Checklist

Subject and description added to both, commit and PR.

Relevant issues have been referenced.

This change includes docs.

This change includes unit tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

nunnatsa · 2023-12-05T11:50:50Z

/test e2e-aws

nunnatsa · 2023-12-05T13:39:51Z

/test e2e-aws
/test e2e-kubevirt-aws-ovn

nunnatsa · 2023-12-05T19:34:10Z

/test e2e-aws
/test e2e-kubevirt-aws-ovn

davidvossel

/lgtm
/approve

openshift-ci · 2023-12-05T22:20:42Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: davidvossel, nunnatsa

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [davidvossel]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci-robot · 2023-12-05T23:17:54Z

/retest-required

Remaining retests: 0 against base HEAD 7f8fcfb and 2 for PR HEAD 5d4ed3f in total

nunnatsa · 2023-12-06T09:54:39Z

/retest-required

nunnatsa · 2023-12-06T13:39:11Z

/retest-required

openshift-ci-robot · 2023-12-06T15:44:39Z

/retest-required

Remaining retests: 0 against base HEAD 39a0c45 and 1 for PR HEAD 5d4ed3f in total

sjenning · 2023-12-06T20:33:50Z

/retest-required

davidvossel · 2023-12-06T21:52:34Z

/cherry-pick release-4.14

openshift-cherrypick-robot · 2023-12-06T21:52:37Z

@davidvossel: once the present PR merges, I will cherry-pick it on top of release-4.14 in a new PR and assign it to you.

Details

In response to this:

/cherry-pick release-4.14

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2023-12-06T21:57:13Z

@nunnatsa: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-ibmcloud-roks	`5d4ed3f`	link	false	`/test e2e-ibmcloud-roks`
ci/prow/e2e-ibmcloud-iks	`5d4ed3f`	link	false	`/test e2e-ibmcloud-iks`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-ci-robot · 2023-12-07T01:20:26Z

/retest-required

Remaining retests: 0 against base HEAD 46ac206 and 0 for PR HEAD 5d4ed3f in total

openshift-ci-robot · 2023-12-07T07:01:44Z

@nunnatsa: Jira Issue OCPBUGS-23126: All pull requests linked via external trackers have merged:

openshift/hypershift#3234

Jira Issue OCPBUGS-23126 has been moved to the MODIFIED state.

Details

In response to this:

What this PR does / why we need it

A user destroying a HostedCluster can cause the HostedCluster to hang indefinitely if the destroy command times out during execution.

This is due to the hcp cli placing a finalizer on the HostedCluster during deletion which the cli tool later removes after waiting for some clean up actions to occur. If a user cancels the hcp destroy cluster command (or the command times out) while the cli is waiting for cleanup, then the HostedCluster will hang indefinitely with a DeletionTimestamp != nil.

This PR only performs the additional cleaning logic, including the setting and removing the special finalizer, for platform that actually creates additional resources. Also, this PR sets the hostedCluster as the owner for these secrets, so they will be deleted with the HostedCluster.

Which issue(s) this PR fixes

Fixes #OCPBUGS-23126

Checklist

Subject and description added to both, commit and PR.

Relevant issues have been referenced.

This change includes docs.

This change includes unit tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-cherrypick-robot · 2023-12-07T07:02:40Z

@davidvossel: #3234 failed to apply on top of branch "release-4.14":

Applying: Fix a bug on deletion of a hostedcluster
Using index info to reconstruct a base tree...
M	api/fixtures/example.go
M	cmd/cluster/core/create.go
M	cmd/cluster/core/destroy.go
M	hypershift-operator/controllers/hostedcluster/hostedcluster_controller.go
M	hypershift-operator/controllers/hostedcluster/hostedcluster_controller_test.go
Falling back to patching base and 3-way merge...
Auto-merging hypershift-operator/controllers/hostedcluster/hostedcluster_controller_test.go
Auto-merging hypershift-operator/controllers/hostedcluster/hostedcluster_controller.go
CONFLICT (content): Merge conflict in hypershift-operator/controllers/hostedcluster/hostedcluster_controller.go
Auto-merging cmd/cluster/core/destroy.go
CONFLICT (content): Merge conflict in cmd/cluster/core/destroy.go
Auto-merging cmd/cluster/core/create.go
Auto-merging api/fixtures/example.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 Fix a bug on deletion of a hostedcluster
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

Details

In response to this:

/cherry-pick release-4.14

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

PR openshift#3234 used the wrong HostedCluster annotation to check if the cluster may be considered as already deleted. This PR fixes the annotation name to `"hypershift.openshift.io/destroy-grace-period"` Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>

openshift-bot · 2023-12-07T08:27:28Z

[ART PR BUILD NOTIFIER]

This PR has been included in build ose-hypershift-container-v4.15.0-202312070751.p0.g5b9c536.assembly.stream for distgit hypershift.
All builds following this will include this PR.

A user destroying a HostedCluster can cause the HostedCluster to hang indefinitely if the destroy command times out during execution. This is due to the hcp cli placing a finalizer on the HostedCluster during deletion which the cli tool later removes after waiting for some clean up actions to occur. If a user cancels the hcp destroy cluster command (or the command times out) while the cli is waiting for cleanup, then the HostedCluster will hang indefinitely with a DeletionTimestamp != nil. This PR only performs the additional cleaning logic, including the setting and removing the special finalizer, for platform that actually creates additional resources. Also, this PR sets the hostedCluster as the owner for these secrets, so they will be deleted with the HostedCluster. This PR is a manual cherry-pick of openshift#3234 Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>

openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 26, 2023

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Nov 26, 2023

openshift-ci-robot added the jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. label Nov 26, 2023

openshift-ci Bot added the do-not-merge/needs-area label Nov 26, 2023

openshift-ci Bot requested review from csrwng and hasueki November 26, 2023 13:50

openshift-ci Bot added area/cli Indicates the PR includes changes for CLI and removed do-not-merge/needs-area labels Nov 26, 2023

nunnatsa force-pushed the fix-OCPBUGS-23126 branch from 8d763bd to 72be1af Compare November 26, 2023 21:06

openshift-ci Bot added the area/hypershift-operator Indicates the PR includes changes for the hypershift operator and API - outside an OCP release label Nov 26, 2023

nunnatsa force-pushed the fix-OCPBUGS-23126 branch from 72be1af to 0158c40 Compare November 27, 2023 06:18

nunnatsa changed the title ~~WIP: OCPBUGS-23126: Fix a bug on deletion of a hostedcluster~~ OCPBUGS-23126: Fix a bug on deletion of a hostedcluster Nov 27, 2023

openshift-ci Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 27, 2023

openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Nov 27, 2023

openshift-ci Bot requested a review from LiangquanLi930 November 27, 2023 07:19

nunnatsa force-pushed the fix-OCPBUGS-23126 branch from 0158c40 to 9bd5da9 Compare November 28, 2023 13:04

nunnatsa force-pushed the fix-OCPBUGS-23126 branch 5 times, most recently from 4bdec0a to 7129c9e Compare November 30, 2023 12:00

nunnatsa force-pushed the fix-OCPBUGS-23126 branch from 21e982c to 5d4ed3f Compare December 5, 2023 09:55

davidvossel reviewed Dec 5, 2023

View reviewed changes

openshift-ci Bot assigned davidvossel Dec 5, 2023

openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Dec 5, 2023

openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 5, 2023

openshift-merge-bot Bot merged commit 5b9c536 into openshift:main Dec 7, 2023

nunnatsa deleted the fix-OCPBUGS-23126 branch December 7, 2023 07:10

nunnatsa mentioned this pull request Dec 7, 2023

NO-JIRA: Fix wrong annotation on cluster deletion #3289

Merged

1 of 4 tasks

nunnatsa mentioned this pull request Dec 7, 2023

[Release 4.14] OCPBUGS-24556: Fix a bug on deletion of a hostedcluster #3290

Merged

3 of 4 tasks

muraee mentioned this pull request Apr 17, 2024

OCPBUGS-32228: Fix failure to create a second hostedcluster in the same namespace #3900

Merged

4 tasks

coderabbitai Bot mentioned this pull request Aug 18, 2026

OCPBUGS-111927: fix(destroy): strip finalizers from all namespaced content before removing control plane namespace #9334

Closed

3 of 4 tasks

Conversation

nunnatsa commented Nov 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does / why we need it

Which issue(s) this PR fixes

Checklist

Uh oh!

openshift-ci-robot commented Nov 26, 2023

Uh oh!

nunnatsa commented Nov 26, 2023

Uh oh!

nunnatsa commented Nov 27, 2023

Uh oh!

openshift-ci-robot commented Nov 27, 2023

Uh oh!

nunnatsa commented Nov 27, 2023

Uh oh!

openshift-ci-robot commented Nov 27, 2023

Uh oh!

openshift-ci-robot commented Nov 28, 2023

What this PR does / why we need it

Which issue(s) this PR fixes

Checklist

Uh oh!

openshift-ci-robot commented Nov 28, 2023

What this PR does / why we need it

Which issue(s) this PR fixes

Checklist

Uh oh!

nunnatsa commented Dec 5, 2023

Uh oh!

nunnatsa commented Dec 5, 2023

Uh oh!

nunnatsa commented Dec 5, 2023

Uh oh!

davidvossel left a comment

Choose a reason for hiding this comment

Uh oh!

openshift-ci Bot commented Dec 5, 2023

Uh oh!

openshift-ci-robot commented Dec 5, 2023

Uh oh!

nunnatsa commented Dec 6, 2023

Uh oh!

nunnatsa commented Dec 6, 2023

Uh oh!

openshift-ci-robot commented Dec 6, 2023

Uh oh!

sjenning commented Dec 6, 2023

Uh oh!

davidvossel commented Dec 6, 2023

Uh oh!

openshift-cherrypick-robot commented Dec 6, 2023

Uh oh!

openshift-ci Bot commented Dec 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Dec 7, 2023

Uh oh!

openshift-ci-robot commented Dec 7, 2023

What this PR does / why we need it

Which issue(s) this PR fixes

Checklist

Uh oh!

openshift-cherrypick-robot commented Dec 7, 2023

Uh oh!

openshift-bot commented Dec 7, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

nunnatsa commented Nov 26, 2023 •

edited

Loading

openshift-ci Bot commented Dec 6, 2023 •

edited

Loading