Skip to content

OCPBUGS-127110: build hypershift-tests from branch PR instead of main - #85648

Closed
jparrill wants to merge 1 commit into
openshift:mainfrom
jparrill:fix-4-22-blocker-v3
Closed

jparrill wants to merge 1 commit into
openshift:mainfrom
jparrill:fix-4-22-blocker-v3

Conversation

@jparrill

@jparrill jparrill commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Build hypershift-tests from Dockerfile.e2e on release branches 4.19-5.0 instead of pulling tag: latest from main
  • Fixes CLI/operator version mismatch in upgrade test jobs (e2e-aws-upgrade-hypershift-operator)
  • Actively breaking release-4.22 CI; latent bug on 4.19-4.21 and 5.0

Root Cause

The hypershift-tests image was defined in base_images with tag: latest, resolving to main (5.1). The test binary contains the hypershift install CLI, which embeds CRDs from its own version. During upgrade tests, the 5.1 CLI applies 5.1 CRDs (CAPI v1beta2 storage version + conversion webhook at /convert) but deploys an operator image from the PR branch (e.g., 4.22) that does not serve /convert. Any CAPI write operation (DELETE MachineDeployment during teardown) triggers a conversion webhook call that returns 404.

Evidence from Prow job 210193874397482598:

  • Post-upgrade RS annotation: install-cli-version: Latest supported OCP: 5.1.0
  • Post-upgrade operator pod log: Latest supported OCP: 4.22.0
  • 913 conversion webhook errors starting at teardown

Changes

For each release branch (4.19, 4.20, 4.21, 4.22, 5.0):

  1. Remove hypershift-tests from base_images (was tag: latest from main)
  2. Add Dockerfile.e2e build in images section (builds from PR source)

Generated job files updated via make jobs.

Test Plan

  • CI presubmit pull-ci-openshift-hypershift-release-4.22-e2e-aws-upgrade-hypershift-operator passes
  • Verify hypershift-tests image is built from PR branch, not main

Ref: OCPBUGS-127110

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Updated hypershift-tests builds on release branches 4.19 through 5.0.
  • Replaced the latest base image from main with branch-local Dockerfile.e2e builds.
  • Prevents CLI and operator version mismatches in e2e-aws-upgrade-hypershift-operator jobs.
  • Regenerated the affected CI job configurations.
  • Addresses OCPBUGS-127110.

@openshift-ci-robot openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. labels Sep 22, 2026
@openshift-ci-robot

openshift-ci-robot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

@jparrill: This pull request references Jira Issue OCPBUGS-127110, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (5.1.0) matches configured target version for branch (5.1.0)
  • bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

  • Build hypershift-tests from Dockerfile.e2e on release branches 4.19-5.0 instead of pulling tag: latest from main
  • Fixes CLI/operator version mismatch in upgrade test jobs (e2e-aws-upgrade-hypershift-operator)
  • Actively breaking release-4.22 CI; latent bug on 4.19-4.21 and 5.0

Root Cause

The hypershift-tests image was defined in base_images with tag: latest, resolving to main (5.1). The test binary contains the hypershift install CLI, which embeds CRDs from its own version. During upgrade tests, the 5.1 CLI applies 5.1 CRDs (CAPI v1beta2 storage version + conversion webhook at /convert) but deploys an operator image from the PR branch (e.g., 4.22) that does not serve /convert. Any CAPI write operation (DELETE MachineDeployment during teardown) triggers a conversion webhook call that returns 404.

Evidence from Prow job 210193874397482598:

  • Post-upgrade RS annotation: install-cli-version: Latest supported OCP: 5.1.0
  • Post-upgrade operator pod log: Latest supported OCP: 4.22.0
  • 913 conversion webhook errors starting at teardown

Changes

For each release branch (4.19, 4.20, 4.21, 4.22, 5.0):

  1. Remove hypershift-tests from base_images (was tag: latest from main)
  2. Add Dockerfile.e2e build in images section (builds from PR source)

Generated job files updated via make jobs.

Test Plan

  • CI presubmit pull-ci-openshift-hypershift-release-4.22-e2e-aws-upgrade-hypershift-operator passes
  • Verify hypershift-tests image is built from PR branch, not main

Ref: OCPBUGS-127110

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@jparrill: This pull request references Jira Issue OCPBUGS-127110, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (5.1.0) matches configured target version for branch (5.1.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
Details

In response to this:

Summary

  • Build hypershift-tests from Dockerfile.e2e on release branches 4.19-5.0 instead of pulling tag: latest from main
  • Fixes CLI/operator version mismatch in upgrade test jobs (e2e-aws-upgrade-hypershift-operator)
  • Actively breaking release-4.22 CI; latent bug on 4.19-4.21 and 5.0

Root Cause

The hypershift-tests image was defined in base_images with tag: latest, resolving to main (5.1). The test binary contains the hypershift install CLI, which embeds CRDs from its own version. During upgrade tests, the 5.1 CLI applies 5.1 CRDs (CAPI v1beta2 storage version + conversion webhook at /convert) but deploys an operator image from the PR branch (e.g., 4.22) that does not serve /convert. Any CAPI write operation (DELETE MachineDeployment during teardown) triggers a conversion webhook call that returns 404.

Evidence from Prow job 210193874397482598:

  • Post-upgrade RS annotation: install-cli-version: Latest supported OCP: 5.1.0
  • Post-upgrade operator pod log: Latest supported OCP: 4.22.0
  • 913 conversion webhook errors starting at teardown

Changes

For each release branch (4.19, 4.20, 4.21, 4.22, 5.0):

  1. Remove hypershift-tests from base_images (was tag: latest from main)
  2. Add Dockerfile.e2e build in images section (builds from PR source)

Generated job files updated via make jobs.

Test Plan

  • CI presubmit pull-ci-openshift-hypershift-release-4.22-e2e-aws-upgrade-hypershift-operator passes
  • Verify hypershift-tests image is built from PR branch, not main

Ref: OCPBUGS-127110

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai

coderabbitai Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

We couldn't safely recover the incremental review. No full review was started, and the last reviewed checkpoint was preserved. Retry later, or explicitly request a full review by commenting @coderabbitai full review.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: cb1940e4-e1bf-4055-8136-8313013b29bc

📥 Commits

Reviewing files that changed from the base of the PR and between 9b33ad9 and b778fcc.

⛔ Files ignored due to path filters (11)
  • ci-operator/jobs/openshift/hypershift/openshift-hypershift-release-4.19-postsubmits.yaml is excluded by !ci-operator/jobs/**
  • ci-operator/jobs/openshift/hypershift/openshift-hypershift-release-4.19-presubmits.yaml is excluded by !ci-operator/jobs/**
  • ci-operator/jobs/openshift/hypershift/openshift-hypershift-release-4.20-postsubmits.yaml is excluded by !ci-operator/jobs/**
  • ci-operator/jobs/openshift/hypershift/openshift-hypershift-release-4.20-presubmits.yaml is excluded by !ci-operator/jobs/**
  • ci-operator/jobs/openshift/hypershift/openshift-hypershift-release-4.21-postsubmits.yaml is excluded by !ci-operator/jobs/**
  • ci-operator/jobs/openshift/hypershift/openshift-hypershift-release-4.21-presubmits.yaml is excluded by !ci-operator/jobs/**
  • ci-operator/jobs/openshift/hypershift/openshift-hypershift-release-4.22-periodics.yaml is excluded by !ci-operator/jobs/**
  • ci-operator/jobs/openshift/hypershift/openshift-hypershift-release-4.22-postsubmits.yaml is excluded by !ci-operator/jobs/**
  • ci-operator/jobs/openshift/hypershift/openshift-hypershift-release-4.22-presubmits.yaml is excluded by !ci-operator/jobs/**
  • ci-operator/jobs/openshift/hypershift/openshift-hypershift-release-5.0-postsubmits.yaml is excluded by !ci-operator/jobs/**
  • ci-operator/jobs/openshift/hypershift/openshift-hypershift-release-5.0-presubmits.yaml is excluded by !ci-operator/jobs/**
📒 Files selected for processing (5)
  • ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.19.yaml
  • ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.20.yaml
  • ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.21.yaml
  • ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.22.yaml
  • ci-operator/config/openshift/hypershift/openshift-hypershift-release-5.0.yaml

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.


Walkthrough

The five Hypershift release configurations now build hypershift-tests from Dockerfile.e2e instead of using a hypershift-tests base image.

Changes

Hypershift test image build configuration

Layer / File(s) Summary
Replace the base image with a Dockerfile build
ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.19.yaml, ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.20.yaml, ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.21.yaml, ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.22.yaml, ci-operator/config/openshift/hypershift/openshift-hypershift-release-5.0.yaml
Each configuration removes the hypershift-tests base image entry and adds a Dockerfile.e2e build that produces hypershift-tests.

Priority: ➖ Normal

Estimated code review effort: 1 (Trivial) | ~5 minutes

Change: Bug fix

Suggested reviewers: mgencur

Merge Risk: ⚪ Minimal · up to b778f

The release jobs will build hypershift-tests from the branch-local Dockerfile.e2e, avoiding the prior floating-image version mismatch. No concrete production-impacting risk remains identified, so the change is ready to merge.

🚥 Pre-merge checks | ✅ 15
✅ Passed checks (15 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: it builds hypershift-tests from the pull request branch instead of main.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed PASS. The PR changes only CI YAML configuration and generated job files. The diff removes the hypershift-tests base image and adds Dockerfile.e2e build references. No changed file contains Ginkgo …
Test Structure And Quality ✅ Passed PASS: The pull request changes only 16 YAML files. The diff updates ci-operator image definitions and generated job file references for Dockerfile.e2e. It adds no Ginkgo test code, It blocks, resource…
Microshift Test Compatibility ✅ Passed PASS: The pull-request range changes only 16 YAML files under ci-operator/config and ci-operator/jobs. It adds no Go or other test source files, and the added lines contain no Ginkgo declarations …
Single Node Openshift (Sno) Test Compatibility ✅ Passed The pull request changes only YAML CI configuration and generated job files. The authoritative diff contains no new or modified Ginkgo e2e test source, so the SNO multi-node compatibility check does n…
Topology-Aware Scheduling Compatibility ✅ Passed PASS. The PR changes only CI image configuration and generated Prow job checkout lists. The config adds Dockerfile.e2e as the hypershift-tests image source and removes the tag: latest base image…
Ote Binary Stdout Contract ✅ Passed PASS — The pull request changes only CI YAML configuration and generated job metadata. It adds Dockerfile.e2e image references and removes a base-image reference. It does not change OTE binaries, suit…
Ipv6 And Disconnected Network Test Compatibility ✅ Passed PASS: The pull request changes only 16 CI YAML files. It adds Dockerfile.e2e image references and removes the hypershift-tests base-image entries. It adds no Ginkgo test declarations or test sourc…
No-Weak-Crypto ✅ Passed The authoritative PR diff changes only CI YAML: it removes the hypershift-tests base-image entries, adds Dockerfile.e2e image targets, and adds Dockerfile.e2e path triggers to generated jobs. No…
Container-Privileges ✅ Passed PASS: The PR changes CI image configuration and generated sparse-checkout lists only. The authoritative diff adds dockerfile_path: Dockerfile.e2e, to: hypershift-tests, and Dockerfile.e2e checko…
No-Sensitive-Data-In-Logs ✅ Passed The pull request adds only Dockerfile.e2e image-build references and removes the hypershift-tests base-image entries. The generated job changes only add Dockerfile.e2e to required input files. N…
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR

Comment @coderabbitai help to get the list of available commands.

@openshift-ci
openshift-ci Bot requested review from csrwng and sdminonne September 22, 2026 10:25
@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 22, 2026
@jparrill

jparrill commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor Author

Context: Why this matters for the HO upgrade test

The hypershift-tests image contains the hypershift install CLI binary, which embeds CRDs from the version it was compiled with. In the e2e-aws-upgrade-hypershift-operator job, this CLI runs hypershift install to perform the upgrade, applying its embedded CRDs to the management cluster before deploying the operator image from the PR.

When hypershift-tests comes from base_images: tag: latest (main), the CLI version diverges from the operator version on release branches. This creates a mismatch:

CLI (from test binary) → applies CRDs from main/5.1
Operator (from PR)     → expects CRDs from its own version (e.g., 4.22)

The concrete failure on 4.22: main's CLI sets CAPI v1beta2 as the storage version and configures a conversion webhook at /convert. The 4.22 operator doesn't serve /convert (CAPI v1beta2 support was added in 5.0). Any CAPI write operation (e.g., DELETE during teardown) triggers a conversion call → 404 → teardown hangs → test timeout.

For the HO upgrade test specifically, the test binary (CLI) and operator image must come from the same source — otherwise CRDs and operator capabilities diverge. Building hypershift-tests from Dockerfile.e2e in the PR source guarantees this.

This was latent on 5.0 (works by coincidence because 5.0 serves /convert), but will break there too as main diverges further.

@jparrill
jparrill force-pushed the fix-4-22-blocker-v3 branch 3 times, most recently from dee9f61 to e86e816 Compare September 22, 2026 11:43
The e2e-aws-upgrade-hypershift-operator test was pulling hypershift-tests
from main via base_images, causing a CLI/operator version mismatch when
the test binary applied newer CRDs to an older operator (e.g., 5.1 CRDs
on 4.22 operator). This broke the CAPIv1beta1→v1beta2 migration path.

Create a dedicated hypershift-aws-e2e-nested-upgrade workflow that
overrides the test step's `from:` to use hypershift-tests-pr (built from
PR source via Dockerfile.e2e). Non-upgrade jobs continue using
hypershift-tests from main, unchanged.

Applied to release branches: 4.19, 4.20, 4.21, 4.22, 5.0, 5.1, 5.2.

Ref: OCPBUGS-127110

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Juan Manuel Parrilla Madrid <jparrill@redhat.com>
@mgencur

mgencur commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

/lgtm

@jparrill

Copy link
Copy Markdown
Contributor Author

/pj-rehearse ack

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@jparrill: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-merge-bot openshift-merge-bot Bot added the rehearsals-ack Signifies that rehearsal jobs have been acknowledged label Sep 22, 2026
@openshift-ci

openshift-ci Bot commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jparrill, mgencur

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Sep 22, 2026
@openshift-merge-bot openshift-merge-bot Bot removed the rehearsals-ack Signifies that rehearsal jobs have been acknowledged label Sep 22, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

[REHEARSALNOTIFIER]
@jparrill: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name Repo Type Reason
pull-ci-openshift-hypershift-release-4.20-e2e-aks openshift/hypershift presubmit Ci-operator config changed
pull-ci-openshift-hypershift-release-4.20-e2e-aws openshift/hypershift presubmit Ci-operator config changed
pull-ci-openshift-hypershift-release-4.20-e2e-aws-autonode openshift/hypershift presubmit Ci-operator config changed
pull-ci-openshift-hypershift-release-4.20-e2e-aws-external-oidc openshift/hypershift presubmit Ci-operator config changed
pull-ci-openshift-hypershift-release-4.20-e2e-aws-metrics openshift/hypershift presubmit Ci-operator config changed
pull-ci-openshift-hypershift-release-4.20-e2e-aws-minimal openshift/hypershift presubmit Ci-operator config changed
pull-ci-openshift-hypershift-release-4.20-e2e-aws-override openshift/hypershift presubmit Ci-operator config changed
pull-ci-openshift-hypershift-release-4.20-e2e-aws-techpreview openshift/hypershift presubmit Ci-operator config changed
pull-ci-openshift-hypershift-release-4.20-e2e-aws-upgrade-hypershift-operator openshift/hypershift presubmit Ci-operator config changed
pull-ci-openshift-hypershift-release-4.20-e2e-azure-aks-external-oidc openshift/hypershift presubmit Ci-operator config changed
pull-ci-openshift-hypershift-release-4.20-e2e-azure-aks-ovn-conformance openshift/hypershift presubmit Ci-operator config changed
pull-ci-openshift-hypershift-release-4.20-e2e-conformance openshift/hypershift presubmit Ci-operator config changed
pull-ci-openshift-hypershift-release-4.20-e2e-kubevirt-aws-ovn openshift/hypershift presubmit Ci-operator config changed
pull-ci-openshift-hypershift-release-4.20-e2e-kubevirt-aws-ovn-reduced openshift/hypershift presubmit Ci-operator config changed
pull-ci-openshift-hypershift-release-4.20-e2e-kubevirt-azure-ovn openshift/hypershift presubmit Ci-operator config changed
pull-ci-openshift-hypershift-release-4.20-e2e-kubevirt-metal-conformance openshift/hypershift presubmit Ci-operator config changed
pull-ci-openshift-hypershift-release-4.20-images openshift/hypershift presubmit Ci-operator config changed
pull-ci-openshift-hypershift-release-4.20-security openshift/hypershift presubmit Ci-operator config changed
pull-ci-openshift-hypershift-release-4.20-unit openshift/hypershift presubmit Ci-operator config changed
pull-ci-openshift-hypershift-release-4.20-verify openshift/hypershift presubmit Ci-operator config changed
pull-ci-openshift-hypershift-release-4.20-verify-deps openshift/hypershift presubmit Ci-operator config changed
pull-ci-openshift-hypershift-release-4.21-e2e-agent-connected-ovn-ipv4-metal-backuprestore openshift/hypershift presubmit Ci-operator config changed
pull-ci-openshift-hypershift-release-4.21-e2e-aks openshift/hypershift presubmit Ci-operator config changed
pull-ci-openshift-hypershift-release-4.21-e2e-aks-override openshift/hypershift presubmit Ci-operator config changed
pull-ci-openshift-hypershift-release-4.21-e2e-aws openshift/hypershift presubmit Ci-operator config changed

A total of 593 jobs have been affected by this change. The above listing is non-exhaustive and limited to 25 jobs.

A full list of affected jobs can be found here

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@openshift-ci

openshift-ci Bot commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

@jparrill: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@devguyio

Copy link
Copy Markdown
Contributor

/hold

@devguyio

Copy link
Copy Markdown
Contributor

This is the wrong approach. Explaining on slack.

@openshift-ci openshift-ci Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 22, 2026
@devguyio

Copy link
Copy Markdown
Contributor

The following is the desired behavior for the upgrade hypershift operator:

  1. HyperShift Image:
    1. pre-upgrade (aka initial hypershift operator image): integration stream of the branch. i.e. latest and release-x.y .
    2. upgrade target: image from PR under test. i.e. main and release-x.y
  2. HyperShift cli:
    1. hypershift image from PR under test, i.e. main and release-x.y
  3. test binary:
    1. hypershift image latest. i.e. main.

The expectations are:

  1. hypershift-install:
    1. The script extracts the hypershift cli from the hypershift operator image.
      1. This means that if the hypershift operator image is set to release-4.22 and INSTALL_FROM_LATEST: "true" is set is in here, the hypershift-install workflow will extract the hypershift cli from the release-4.22 hypershift operator image is you see here.
  2. hypershift-test binary:
    1. Will accommodate to running against all releases.
    2. If there's a behavior that needs to happen on main only, it should gate that.
    3. If crds need to be installed for main only, then it needs to gate that.

@jparrill

Copy link
Copy Markdown
Contributor Author

/close

In favor of openshift/hypershift#9736 (comment)

@openshift-ci openshift-ci Bot closed this Sep 23, 2026
@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@jparrill: This pull request references Jira Issue OCPBUGS-127110. The bug has been updated to no longer refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

  • Build hypershift-tests from Dockerfile.e2e on release branches 4.19-5.0 instead of pulling tag: latest from main
  • Fixes CLI/operator version mismatch in upgrade test jobs (e2e-aws-upgrade-hypershift-operator)
  • Actively breaking release-4.22 CI; latent bug on 4.19-4.21 and 5.0

Root Cause

The hypershift-tests image was defined in base_images with tag: latest, resolving to main (5.1). The test binary contains the hypershift install CLI, which embeds CRDs from its own version. During upgrade tests, the 5.1 CLI applies 5.1 CRDs (CAPI v1beta2 storage version + conversion webhook at /convert) but deploys an operator image from the PR branch (e.g., 4.22) that does not serve /convert. Any CAPI write operation (DELETE MachineDeployment during teardown) triggers a conversion webhook call that returns 404.

Evidence from Prow job 210193874397482598:

  • Post-upgrade RS annotation: install-cli-version: Latest supported OCP: 5.1.0
  • Post-upgrade operator pod log: Latest supported OCP: 4.22.0
  • 913 conversion webhook errors starting at teardown

Changes

For each release branch (4.19, 4.20, 4.21, 4.22, 5.0):

  1. Remove hypershift-tests from base_images (was tag: latest from main)
  2. Add Dockerfile.e2e build in images section (builds from PR source)

Generated job files updated via make jobs.

Test Plan

  • CI presubmit pull-ci-openshift-hypershift-release-4.22-e2e-aws-upgrade-hypershift-operator passes
  • Verify hypershift-tests image is built from PR branch, not main

Ref: OCPBUGS-127110

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Updated hypershift-tests builds on release branches 4.19 through 5.0.
  • Replaced the latest base image from main with branch-local Dockerfile.e2e builds.
  • Prevents CLI and operator version mismatches in e2e-aws-upgrade-hypershift-operator jobs.
  • Regenerated the affected CI job configurations.
  • Addresses OCPBUGS-127110.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci

openshift-ci Bot commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

@jparrill: Closed this PR.

Details

In response to this:

/close

In favor of openshift/hypershift#9736 (comment)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants