Skip to content

OCPBUGS-74960: Add CPO image override and private cluster SG cleanup verification - #76891

Closed
zhfeng wants to merge 5 commits into
openshift:mainfrom
zhfeng:verify-OCPBUGS-74960-sg-cleanup
Closed

zhfeng wants to merge 5 commits into
openshift:mainfrom
zhfeng:verify-OCPBUGS-74960-sg-cleanup

Conversation

@zhfeng

@zhfeng zhfeng commented Mar 26, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds a periodic CI job e2e-aws-private-sg-cleanup to verify the OCPBUGS-74960 fix prevents VPC endpoint security group leaks when private HostedClusters are deleted.

HyperShift PR: openshift/hypershift#7868
Jira: OCPBUGS-74960

What this job does

  1. Baseline (hypershift-aws-sg-baseline): Records all *-vpce-private-router security groups before the test
  2. Test (hypershift-aws-run-e2e-external): Runs TestCreateClusterPrivate with the CPO image from PR Use namespace resource rather than ProjectRequest to create NS #7868 (via --e2e.control-plane-operator-image override)
  3. Verify (hypershift-aws-verify-sg-cleanup): After cluster deletion, checks for orphaned SGs by comparing against baseline and verifying each new SG's VPC for active endpoints

Rehearsal Results (Iteration 4)

Step Result Duration
ipi-install-rbac SUCCESS -
create-management-cluster SUCCESS -
hypershift-install SUCCESS -
hypershift-aws-sg-baseline SUCCESS -
hypershift-aws-run-e2e-external FAILED (transient) 35m12s
hypershift-aws-verify-sg-cleanup SUCCESS 3m21s
dump-management-cluster SUCCESS 2m9s
destroy-management-cluster SUCCESS 6m7s

SG Verify Output

  • Baseline: 41 vpce-private-router SGs
  • After test: 50 vpce-private-router SGs (44 new since baseline)
  • Orphaned: 0 - all 44 new SGs belong to VPCs with active endpoints (from other parallel CI jobs)
  • Our HC private-fs4xj SG: properly cleaned up (not in orphaned list)
  • Result: [PASS] All new SGs belong to VPCs with active endpoints / RESULT: ALL CHECKS PASSED

E2e Test Note

The e2e test step failed with a transient controlPlaneVersion has no desired image condition validation timeout (client rate limiter Wait returned an error: context deadline exceeded). The cluster actually deployed successfully (nodes ready in 10m48s, rollout in 5m6s) and was destroyed cleanly. This is a known CI rate-limiter issue unrelated to the fix.

CPO Image Override Confirmed

The CPO image from PR #7868 was correctly applied at runtime:

--e2e.control-plane-operator-image=quay.io/redhat-user-workloads/crt-redhat-acm-tenant/control-plane-operator-main:on-pr-4ef1d5e1d01b7f3a277f6c35968d50e6b4ff39bf

Rehearsal Job

Prow Job Link

New CI Step Registry Components

  • hypershift-aws-sg-baseline - records SG baseline before test
  • hypershift-aws-verify-sg-cleanup - verifies no orphaned SGs after cluster deletion
  • hypershift-aws-e2e-external-sg-verify workflow - orchestrates the full verification

/cc @openshift/hypershift-team

@zhfeng

zhfeng commented Mar 26, 2026

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-hypershift-release-4.23-periodics-e2e-aws-private-sg-cleanup

@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 26, 2026
@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Mar 26, 2026
@openshift-ci

openshift-ci Bot commented Mar 26, 2026

Copy link
Copy Markdown
Contributor

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@openshift-ci-robot openshift-ci-robot added the jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. label Mar 26, 2026
@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@zhfeng: This pull request references Jira Issue OCPBUGS-74960, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.22.0) matches configured target version for branch (4.22.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @zhfeng

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

  • Adds CONTROL_PLANE_OPERATOR_IMAGE env var support to hypershift-aws-run-e2e-external step, enabling CPO image overrides via --e2e.control-plane-operator-image
  • Adds periodic job e2e-aws-private-sg-cleanup that runs TestCreateClusterPrivate with a Konflux-built CPO image from openshift/hypershift#7868 to verify the OCPBUGS-74960 fix

Context

OCPBUGS-74960: Security groups for VPC endpoints (*-vpce-private-router) are leaked ~40% of the time during HCP cluster deletion due to:

  1. DependencyViolation errors being swallowed during SG deletion
  2. getClients failure after CPO restart causing premature finalizer removal

The fix in hypershift#7868 hardens the deletion path. This job verifies the fix works end-to-end by creating and deleting a private AWS cluster with the fixed CPO image.

Changes

File Change
hypershift-aws-run-e2e-external-ref.yaml Add CONTROL_PLANE_OPERATOR_IMAGE env var
hypershift-aws-run-e2e-external-commands.sh Pass --e2e.control-plane-operator-image when set
openshift-hypershift-release-4.23__periodics.yaml Add e2e-aws-private-sg-cleanup periodic job

Test plan

  • Rehearse via /pj-rehearse periodic-ci-openshift-hypershift-release-4.23-periodics-e2e-aws-private-sg-cleanup
  • Verify TestCreateClusterPrivate passes with the CPO override image
  • Verify no orphaned security groups after cluster deletion

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@zhfeng: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-ci

openshift-ci Bot commented Mar 26, 2026

Copy link
Copy Markdown
Contributor

@openshift-ci-robot: GitHub didn't allow me to request PR reviews from the following users: zhfeng.

Note that only openshift members and repo collaborators can review this PR, and authors cannot review their own PRs.

Details

In response to this:

@zhfeng: This pull request references Jira Issue OCPBUGS-74960, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.22.0) matches configured target version for branch (4.22.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @zhfeng

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

Summary

  • Adds CONTROL_PLANE_OPERATOR_IMAGE env var support to hypershift-aws-run-e2e-external step, enabling CPO image overrides via --e2e.control-plane-operator-image
  • Adds periodic job e2e-aws-private-sg-cleanup that runs TestCreateClusterPrivate with a Konflux-built CPO image from openshift/hypershift#7868 to verify the OCPBUGS-74960 fix

Context

OCPBUGS-74960: Security groups for VPC endpoints (*-vpce-private-router) are leaked ~40% of the time during HCP cluster deletion due to:

  1. DependencyViolation errors being swallowed during SG deletion
  2. getClients failure after CPO restart causing premature finalizer removal

The fix in hypershift#7868 hardens the deletion path. This job verifies the fix works end-to-end by creating and deleting a private AWS cluster with the fixed CPO image.

Changes

File Change
hypershift-aws-run-e2e-external-ref.yaml Add CONTROL_PLANE_OPERATOR_IMAGE env var
hypershift-aws-run-e2e-external-commands.sh Pass --e2e.control-plane-operator-image when set
openshift-hypershift-release-4.23__periodics.yaml Add e2e-aws-private-sg-cleanup periodic job

Test plan

  • Rehearse via /pj-rehearse periodic-ci-openshift-hypershift-release-4.23-periodics-e2e-aws-private-sg-cleanup
  • Verify TestCreateClusterPrivate passes with the CPO override image
  • Verify no orphaned security groups after cluster deletion

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-ci

openshift-ci Bot commented Mar 26, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: zhfeng
Once this PR has been reviewed and has the lgtm label, please assign sjenning for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@zhfeng

zhfeng commented Mar 26, 2026

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-hypershift-release-4.23-periodics-e2e-aws-private-sg-cleanup

@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@zhfeng: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@zhfeng: This pull request references Jira Issue OCPBUGS-74960, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.22.0) matches configured target version for branch (4.22.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @zhfeng

Details

In response to this:

Summary

  • Adds CONTROL_PLANE_OPERATOR_IMAGE env var support to hypershift-aws-run-e2e-external step, enabling CPO image overrides via --e2e.control-plane-operator-image
  • Adds pre/post security group verification steps:
  • hypershift-aws-sg-baseline: records existing *-vpce-private-router SGs before the test
  • hypershift-aws-verify-sg-cleanup: diffs SGs after test to detect orphaned ones
  • Adds hypershift-aws-e2e-external-sg-verify workflow wrapping the existing external e2e workflow with SG verification
  • Adds periodic job e2e-aws-private-sg-cleanup that runs TestCreateClusterPrivate with a Konflux-built CPO image from openshift/hypershift#7868

Context

OCPBUGS-74960: Security groups for VPC endpoints (*-vpce-private-router) are leaked ~40% of the time during HCP cluster deletion due to:

  1. DependencyViolation errors being swallowed during SG deletion
  2. getClients failure after CPO restart causing premature finalizer removal

The fix in hypershift#7868 hardens the deletion path. This job verifies the fix works end-to-end:

  1. Records existing SGs as baseline
  2. Creates a private AWS cluster with the fixed CPO image
  3. Deletes the cluster
  4. Verifies no new orphaned SGs appeared (diff against baseline)

Test plan

  • Rehearse via /pj-rehearse periodic-ci-openshift-hypershift-release-4.23-periodics-e2e-aws-private-sg-cleanup
  • Verify TestCreateClusterPrivate passes with the CPO override image
  • Verify the SG verification step reports [PASS] No orphaned vpce-private-router security groups

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci

openshift-ci Bot commented Mar 26, 2026

Copy link
Copy Markdown
Contributor

@openshift-ci-robot: GitHub didn't allow me to request PR reviews from the following users: zhfeng.

Note that only openshift members and repo collaborators can review this PR, and authors cannot review their own PRs.

Details

In response to this:

@zhfeng: This pull request references Jira Issue OCPBUGS-74960, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.22.0) matches configured target version for branch (4.22.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @zhfeng

In response to this:

Summary

  • Adds CONTROL_PLANE_OPERATOR_IMAGE env var support to hypershift-aws-run-e2e-external step, enabling CPO image overrides via --e2e.control-plane-operator-image
  • Adds pre/post security group verification steps:
  • hypershift-aws-sg-baseline: records existing *-vpce-private-router SGs before the test
  • hypershift-aws-verify-sg-cleanup: diffs SGs after test to detect orphaned ones
  • Adds hypershift-aws-e2e-external-sg-verify workflow wrapping the existing external e2e workflow with SG verification
  • Adds periodic job e2e-aws-private-sg-cleanup that runs TestCreateClusterPrivate with a Konflux-built CPO image from openshift/hypershift#7868

Context

OCPBUGS-74960: Security groups for VPC endpoints (*-vpce-private-router) are leaked ~40% of the time during HCP cluster deletion due to:

  1. DependencyViolation errors being swallowed during SG deletion
  2. getClients failure after CPO restart causing premature finalizer removal

The fix in hypershift#7868 hardens the deletion path. This job verifies the fix works end-to-end:

  1. Records existing SGs as baseline
  2. Creates a private AWS cluster with the fixed CPO image
  3. Deletes the cluster
  4. Verifies no new orphaned SGs appeared (diff against baseline)

Test plan

  • Rehearse via /pj-rehearse periodic-ci-openshift-hypershift-release-4.23-periodics-e2e-aws-private-sg-cleanup
  • Verify TestCreateClusterPrivate passes with the CPO override image
  • Verify the SG verification step reports [PASS] No orphaned vpce-private-router security groups

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@zhfeng

zhfeng commented Mar 26, 2026

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-hypershift-release-4.23-periodics-e2e-aws-private-sg-cleanup

@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@zhfeng: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@zhfeng

zhfeng commented Mar 26, 2026

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-hypershift-release-4.23-periodics-e2e-aws-private-sg-cleanup

@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@zhfeng: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@zhfeng

zhfeng commented Mar 26, 2026

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-hypershift-release-4.23-periodics-e2e-aws-private-sg-cleanup

@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@zhfeng: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-ci-robot

openshift-ci-robot commented Mar 26, 2026 •

Copy link
Copy Markdown
Contributor

@zhfeng: This pull request references Jira Issue OCPBUGS-74960, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.22.0) matches configured target version for branch (4.22.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @zhfeng

Details

In response to this:

Summary

Adds a periodic CI job e2e-aws-private-sg-cleanup to verify the OCPBUGS-74960 fix prevents VPC endpoint security group leaks when private HostedClusters are deleted.

HyperShift PR: openshift/hypershift#7868
Jira: OCPBUGS-74960

What this job does

  1. Baseline (hypershift-aws-sg-baseline): Records all *-vpce-private-router security groups before the test
  2. Test (hypershift-aws-run-e2e-external): Runs TestCreateClusterPrivate with the CPO image from PR Use namespace resource rather than ProjectRequest to create NS #7868 (via --e2e.control-plane-operator-image override)
  3. Verify (hypershift-aws-verify-sg-cleanup): After cluster deletion, checks for orphaned SGs by comparing against baseline and verifying each new SG's VPC for active endpoints

Rehearsal Results (Iteration 4)

Step Result Duration
ipi-install-rbac SUCCESS -
create-management-cluster SUCCESS -
hypershift-install SUCCESS -
hypershift-aws-sg-baseline SUCCESS -
hypershift-aws-run-e2e-external FAILED (transient) 35m12s
hypershift-aws-verify-sg-cleanup SUCCESS 3m21s
dump-management-cluster SUCCESS 2m9s
destroy-management-cluster SUCCESS 6m7s

SG Verify Output

  • Baseline: 41 vpce-private-router SGs
  • After test: 50 vpce-private-router SGs (44 new since baseline)
  • Orphaned: 0 - all 44 new SGs belong to VPCs with active endpoints (from other parallel CI jobs)
  • Our HC private-fs4xj SG: properly cleaned up (not in orphaned list)
  • Result: [PASS] All new SGs belong to VPCs with active endpoints / RESULT: ALL CHECKS PASSED

E2e Test Note

The e2e test step failed with a transient controlPlaneVersion has no desired image condition validation timeout (client rate limiter Wait returned an error: context deadline exceeded). The cluster actually deployed successfully (nodes ready in 10m48s, rollout in 5m6s) and was destroyed cleanly. This is a known CI rate-limiter issue unrelated to the fix.

CPO Image Override Confirmed

The CPO image from PR #7868 was correctly applied at runtime:

--e2e.control-plane-operator-image=quay.io/redhat-user-workloads/crt-redhat-acm-tenant/control-plane-operator-main:on-pr-4ef1d5e1d01b7f3a277f6c35968d50e6b4ff39bf

Rehearsal Job

Prow Job Link

New CI Step Registry Components

  • hypershift-aws-sg-baseline - records SG baseline before test
  • hypershift-aws-verify-sg-cleanup - verifies no orphaned SGs after cluster deletion
  • hypershift-aws-e2e-external-sg-verify workflow - orchestrates the full verification

/cc @openshift/hypershift-team

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci

openshift-ci Bot commented Mar 26, 2026 •

Copy link
Copy Markdown
Contributor

@openshift-ci-robot: GitHub didn't allow me to request PR reviews from the following users: zhfeng.

Note that only openshift members and repo collaborators can review this PR, and authors cannot review their own PRs.

Details

In response to this:

@zhfeng: This pull request references Jira Issue OCPBUGS-74960, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.22.0) matches configured target version for branch (4.22.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @zhfeng

In response to this:

Summary

Adds a periodic CI job e2e-aws-private-sg-cleanup to verify the OCPBUGS-74960 fix prevents VPC endpoint security group leaks when private HostedClusters are deleted.

HyperShift PR: openshift/hypershift#7868
Jira: OCPBUGS-74960

What this job does

  1. Baseline (hypershift-aws-sg-baseline): Records all *-vpce-private-router security groups before the test
  2. Test (hypershift-aws-run-e2e-external): Runs TestCreateClusterPrivate with the CPO image from PR Use namespace resource rather than ProjectRequest to create NS #7868 (via --e2e.control-plane-operator-image override)
  3. Verify (hypershift-aws-verify-sg-cleanup): After cluster deletion, checks for orphaned SGs by comparing against baseline and verifying each new SG's VPC for active endpoints

Rehearsal Results (Iteration 4)

Step Result Duration
ipi-install-rbac SUCCESS -
create-management-cluster SUCCESS -
hypershift-install SUCCESS -
hypershift-aws-sg-baseline SUCCESS -
hypershift-aws-run-e2e-external FAILED (transient) 35m12s
hypershift-aws-verify-sg-cleanup SUCCESS 3m21s
dump-management-cluster SUCCESS 2m9s
destroy-management-cluster SUCCESS 6m7s

SG Verify Output

  • Baseline: 41 vpce-private-router SGs
  • After test: 50 vpce-private-router SGs (44 new since baseline)
  • Orphaned: 0 - all 44 new SGs belong to VPCs with active endpoints (from other parallel CI jobs)
  • Our HC private-fs4xj SG: properly cleaned up (not in orphaned list)
  • Result: [PASS] All new SGs belong to VPCs with active endpoints / RESULT: ALL CHECKS PASSED

E2e Test Note

The e2e test step failed with a transient controlPlaneVersion has no desired image condition validation timeout (client rate limiter Wait returned an error: context deadline exceeded). The cluster actually deployed successfully (nodes ready in 10m48s, rollout in 5m6s) and was destroyed cleanly. This is a known CI rate-limiter issue unrelated to the fix.

CPO Image Override Confirmed

The CPO image from PR #7868 was correctly applied at runtime:

--e2e.control-plane-operator-image=quay.io/redhat-user-workloads/crt-redhat-acm-tenant/control-plane-operator-main:on-pr-4ef1d5e1d01b7f3a277f6c35968d50e6b4ff39bf

Rehearsal Job

Prow Job Link

New CI Step Registry Components

  • hypershift-aws-sg-baseline - records SG baseline before test
  • hypershift-aws-verify-sg-cleanup - verifies no orphaned SGs after cluster deletion
  • hypershift-aws-e2e-external-sg-verify workflow - orchestrates the full verification

/cc @openshift/hypershift-team

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-ci openshift-ci Bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 2, 2026
@zhfeng

zhfeng commented Apr 2, 2026

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-hypershift-release-4.23-periodics-e2e-aws-private-sg-cleanup

@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@zhfeng: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-ci-robot openshift-ci-robot added the rehearsals-ack Signifies that rehearsal jobs have been acknowledged label Apr 2, 2026
@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@zhfeng: job(s): periodic-ci-openshift-hypershift-release-4.23-periodics-e2e-aws-private-sg-cleanup either don't exist or were not found to be affected, and cannot be rehearsed

zhfeng and others added 3 commits April 2, 2026 09:35
Add CONTROL_PLANE_OPERATOR_IMAGE env var to hypershift-aws-run-e2e-external
step to allow overriding the control plane operator image via
--e2e.control-plane-operator-image flag.

Add periodic job e2e-aws-private-sg-cleanup that runs TestCreateClusterPrivate
with a Konflux-built CPO image from openshift/hypershift#7868 to verify the
OCPBUGS-74960 fix: security groups for VPC endpoints are properly cleaned up
during cluster deletion.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add pre-test step (hypershift-aws-sg-baseline) to record existing
vpce-private-router security groups before the test.

Add post-test step (hypershift-aws-verify-sg-cleanup) that compares
current SGs against the baseline to detect orphaned SGs from the test.

Add workflow (hypershift-aws-e2e-external-sg-verify) that wraps the
existing external e2e workflow with these SG verification steps.

Update the periodic job to use the new sg-verify workflow.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The cli image does not have aws CLI or unzip. Switch to
upi-installer which has aws CLI pre-installed.
zhfeng added 2 commits April 2, 2026 09:35
Instead of flagging all new SGs since baseline (which includes SGs
from other parallel tests), check each new SG's VPC for active
endpoints. A truly orphaned SG has no active VPC endpoints in its
VPC, meaning the endpoint was deleted but the SG was not.
@zhfeng
zhfeng force-pushed the verify-OCPBUGS-74960-sg-cleanup branch from ddaa89c to eb41924 Compare April 2, 2026 01:35
@openshift-ci openshift-ci Bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 2, 2026
@zhfeng

zhfeng commented Apr 2, 2026

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-hypershift-release-4.23-periodics-e2e-aws-private-sg-cleanup

@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@zhfeng: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-ci-robot openshift-ci-robot removed the rehearsals-ack Signifies that rehearsal jobs have been acknowledged label Apr 2, 2026
@openshift-ci-robot

Copy link
Copy Markdown
Contributor

[REHEARSALNOTIFIER]
@zhfeng: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name Repo Type Reason
pull-ci-openshift-cloud-credential-operator-master-e2e-hypershift openshift/cloud-credential-operator presubmit Registry content changed
pull-ci-openshift-cloud-credential-operator-release-5.0-e2e-hypershift openshift/cloud-credential-operator presubmit Registry content changed
pull-ci-openshift-cloud-credential-operator-release-4.23-e2e-hypershift openshift/cloud-credential-operator presubmit Registry content changed
pull-ci-openshift-cloud-credential-operator-release-4.22-e2e-hypershift openshift/cloud-credential-operator presubmit Registry content changed
pull-ci-openshift-cloud-credential-operator-release-4.21-e2e-hypershift openshift/cloud-credential-operator presubmit Registry content changed
pull-ci-openshift-cloud-credential-operator-release-4.20-e2e-hypershift openshift/cloud-credential-operator presubmit Registry content changed
pull-ci-openshift-cloud-credential-operator-release-4.19-e2e-hypershift openshift/cloud-credential-operator presubmit Registry content changed
pull-ci-openshift-cloud-credential-operator-release-4.18-e2e-hypershift openshift/cloud-credential-operator presubmit Registry content changed
pull-ci-openshift-cloud-credential-operator-release-4.17-e2e-hypershift openshift/cloud-credential-operator presubmit Registry content changed
pull-ci-openshift-cloud-credential-operator-release-4.16-e2e-hypershift openshift/cloud-credential-operator presubmit Registry content changed
pull-ci-openshift-cloud-credential-operator-release-4.15-e2e-hypershift openshift/cloud-credential-operator presubmit Registry content changed
pull-ci-openshift-cloud-credential-operator-release-4.14-e2e-hypershift openshift/cloud-credential-operator presubmit Registry content changed
pull-ci-openshift-cluster-machine-approver-main-e2e-hypershift-aws openshift/cluster-machine-approver presubmit Registry content changed
pull-ci-openshift-cluster-machine-approver-release-5.0-e2e-hypershift-aws openshift/cluster-machine-approver presubmit Registry content changed
pull-ci-openshift-cluster-machine-approver-release-4.23-e2e-hypershift-aws openshift/cluster-machine-approver presubmit Registry content changed
pull-ci-openshift-cluster-machine-approver-release-4.22-e2e-hypershift-aws openshift/cluster-machine-approver presubmit Registry content changed
pull-ci-openshift-aws-ebs-csi-driver-operator-release-4.14-hypershift-aws-e2e-external openshift/aws-ebs-csi-driver-operator presubmit Registry content changed
pull-ci-openshift-aws-ebs-csi-driver-operator-release-4.13-hypershift-aws-e2e-external openshift/aws-ebs-csi-driver-operator presubmit Registry content changed
pull-ci-openshift-aws-ebs-csi-driver-operator-release-4.12-hypershift-aws-e2e-external openshift/aws-ebs-csi-driver-operator presubmit Registry content changed
pull-ci-openshift-cluster-version-operator-main-e2e-hypershift openshift/cluster-version-operator presubmit Registry content changed
pull-ci-openshift-cluster-version-operator-release-5.0-e2e-hypershift openshift/cluster-version-operator presubmit Registry content changed
pull-ci-openshift-cluster-version-operator-release-4.23-e2e-hypershift openshift/cluster-version-operator presubmit Registry content changed
pull-ci-openshift-cluster-version-operator-release-4.22-e2e-hypershift openshift/cluster-version-operator presubmit Registry content changed
pull-ci-openshift-cluster-version-operator-release-4.21-e2e-hypershift openshift/cluster-version-operator presubmit Registry content changed
pull-ci-openshift-cluster-version-operator-release-4.20-e2e-hypershift openshift/cluster-version-operator presubmit Registry content changed

A total of 278 jobs have been affected by this change. The above listing is non-exhaustive and limited to 25 jobs.

A full list of affected jobs can be found here

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@openshift-ci

openshift-ci Bot commented Apr 2, 2026

Copy link
Copy Markdown
Contributor

@zhfeng: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@zhfeng zhfeng closed this Apr 2, 2026
@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@zhfeng: This pull request references Jira Issue OCPBUGS-74960. The bug has been updated to no longer refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

Adds a periodic CI job e2e-aws-private-sg-cleanup to verify the OCPBUGS-74960 fix prevents VPC endpoint security group leaks when private HostedClusters are deleted.

HyperShift PR: openshift/hypershift#7868
Jira: OCPBUGS-74960

What this job does

  1. Baseline (hypershift-aws-sg-baseline): Records all *-vpce-private-router security groups before the test
  2. Test (hypershift-aws-run-e2e-external): Runs TestCreateClusterPrivate with the CPO image from PR Use namespace resource rather than ProjectRequest to create NS #7868 (via --e2e.control-plane-operator-image override)
  3. Verify (hypershift-aws-verify-sg-cleanup): After cluster deletion, checks for orphaned SGs by comparing against baseline and verifying each new SG's VPC for active endpoints

Rehearsal Results (Iteration 4)

Step Result Duration
ipi-install-rbac SUCCESS -
create-management-cluster SUCCESS -
hypershift-install SUCCESS -
hypershift-aws-sg-baseline SUCCESS -
hypershift-aws-run-e2e-external FAILED (transient) 35m12s
hypershift-aws-verify-sg-cleanup SUCCESS 3m21s
dump-management-cluster SUCCESS 2m9s
destroy-management-cluster SUCCESS 6m7s

SG Verify Output

  • Baseline: 41 vpce-private-router SGs
  • After test: 50 vpce-private-router SGs (44 new since baseline)
  • Orphaned: 0 - all 44 new SGs belong to VPCs with active endpoints (from other parallel CI jobs)
  • Our HC private-fs4xj SG: properly cleaned up (not in orphaned list)
  • Result: [PASS] All new SGs belong to VPCs with active endpoints / RESULT: ALL CHECKS PASSED

E2e Test Note

The e2e test step failed with a transient controlPlaneVersion has no desired image condition validation timeout (client rate limiter Wait returned an error: context deadline exceeded). The cluster actually deployed successfully (nodes ready in 10m48s, rollout in 5m6s) and was destroyed cleanly. This is a known CI rate-limiter issue unrelated to the fix.

CPO Image Override Confirmed

The CPO image from PR #7868 was correctly applied at runtime:

--e2e.control-plane-operator-image=quay.io/redhat-user-workloads/crt-redhat-acm-tenant/control-plane-operator-main:on-pr-4ef1d5e1d01b7f3a277f6c35968d50e6b4ff39bf

Rehearsal Job

Prow Job Link

New CI Step Registry Components

  • hypershift-aws-sg-baseline - records SG baseline before test
  • hypershift-aws-verify-sg-cleanup - verifies no orphaned SGs after cluster deletion
  • hypershift-aws-e2e-external-sg-verify workflow - orchestrates the full verification

/cc @openshift/hypershift-team

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants