CNTRLPLANE-3037,OCPBUGS-78837: Introduce envtest for integration tests by clebs · Pull Request #8016 · openshift/hypershift

clebs · 2026-03-19T16:34:48Z

What this PR does / why we need it:

This PR introduces envtest and replicates the existing TestOnCreateAPIUX e2e test (but not replacing it yet).

Shifting part of the tests away from e2e into integration tests that do not require a full environment will help reduce the burden on our CI.

Checklist:

Subject and description added to both, commit and PR.
Relevant issues have been referenced.
This change includes docs.
This change includes unit tests.

openshift-ci-robot · 2026-03-19T16:34:52Z

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

openshift-ci-robot · 2026-03-19T16:34:53Z

@clebs: This pull request explicitly references no jira issue.

Details

In response to this:

What this PR does / why we need it:

This PR introduces envtest and replicates the existing TestOnCreateAPIUX e2e test (but not replacing it yet).

Shifting part of the tests away from e2e into integration tests that do not require a full environment will help reduce the burden on our CI.

Checklist:

Subject and description added to both, commit and PR.

Relevant issues have been referenced.

This change includes docs.

This change includes unit tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci · 2026-03-19T16:34:54Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

coderabbitai · 2026-03-19T16:34:57Z

Important

Review skipped

Auto reviews are limited based on label configuration.

🚫 Review skipped — only excluded labels are configured. (1)

do-not-merge/work-in-progress

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro

Run ID: 42ad9f1e-04dd-4d56-9924-ec7a2ceef078

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

openshift-ci · 2026-03-19T16:35:18Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: clebs
Once this PR has been reviewed and has the lgtm label, please assign enxebre for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

clebs · 2026-03-19T17:04:56Z

/test all

clebs · 2026-03-19T20:18:18Z

/test all

openshift-ci-robot · 2026-03-20T08:31:48Z

@clebs: This pull request references CNTRLPLANE-3037 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set.

Details

In response to this:

What this PR does / why we need it:

This PR introduces envtest and replicates the existing TestOnCreateAPIUX e2e test (but not replacing it yet).

Shifting part of the tests away from e2e into integration tests that do not require a full environment will help reduce the burden on our CI.

Checklist:

Subject and description added to both, commit and PR.

Relevant issues have been referenced.

This change includes docs.

This change includes unit tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

enxebre · 2026-03-20T08:31:48Z

+	// Start envtest with HyperShift CRDs
+	testEnv := &envtest.Environment{
+		CRDDirectoryPaths: []string{crdDir},
+		// Use TechPreviewNoUpgrade CRDs to include GCP validations


// Use TechPreviewNoUpgrade CRDs to include GCP validations

where is this enforced? the path contains also default CRD version. I assume last one would win and it's circumstantially TechPreview? which is fragile.

we'll also want to install all the capi CRDs.
Basically what --install command with tech preview would apply.

I took the simplest approach here as first try. Now looking closer at the e2e I see I missed the whole filtering logic on assets.

I will use that package the same way the original test does when installing so we can select the feature set to test. Maybe for now with an env var defaulting to TechPreviewNoUpgrade.

enxebre · 2026-03-20T08:33:03Z

+	cd $(TOOLS_DIR); $(GO) build -tags=tools -o $(SETUP_ENVTEST) sigs.k8s.io/controller-runtime/tools/setup-envtest
+
+.PHONY: setup-envtest
+setup-envtest: $(SETUP_ENVTEST) ## Setup envtest binaries (etcd, kube-apiserver)


I think we'll want to make this plumbing more flexible so we can test with different kas API versions

also we'll want to also use ocp releases https://raw.githubusercontent.com/openshift/api/master/envtest-releases.yaml
https://github.com/openshift/api/blob/1f2fa3f09f4ea0d6c2b5a09ba8608b70c41f616f/tests/Makefile#L21-L28

+1 Alberto

I don't care too much about running individual validation tests on older API servers, but I do think we need to at least take all CRDs and install them on the oldest OCP KAS that managed services are currently using (4.15 to my knowledge) and then as we start to enforce a skew policy and they upgrade to newer releases, we can bump that and hopefully get to a point where it is N-3 consistently going forward

we need ability to test against oldest ocp (4.15) and oldest vanilla kube (to be confirmed).

We currently have 4.17+ OCP KAS in the index, shouldn't be too hard to push 4.15 and 4.16

Upstream goes back to 1.28 (which is equiv to 4.15) so hopefully the minimum vanilla is at least that

enxebre · 2026-03-20T08:33:46Z

/area api
/assign @JoelSpeed

JoelSpeed · 2026-03-20T09:51:56Z

In openshift/api we made the choice to leverage YAML based testing for our integration tests. With the structured testing you currently have, you have no visibility into the marshalled content that is actually being sent to the API server. This can mask issues in your APIs (e.g. not having omitempty on a required field) so you may want to either us unstructured or switch to YAML as we do

You'll also want to look at the possibility of testing ratcheting validations. Over time you may want to make validation tighter, and with that you need a way to install older, invalid schemas and check the ratcheting.

It might be worth looking over our test suite for inspiration

enxebre · 2026-03-20T11:48:37Z

Let's adjust the PR to cover #8016 (comment)
Get that gating on CI.
Then follow up to address #8016 (comment)

- Add an envtest setup goal on the Makefile - Add a new unit test using envtest to cover the same as the create cluster e2e test. Signed-off-by: Borja Clemente <bclement@redhat.com>

Introducing envtest requires adding new dependencies, which are being vendored in a separate commit to ease review. Signed-off-by: Borja Clemente <bclement@redhat.com>

JoelSpeed · 2026-03-21T10:44:40Z


+# Setup envtest for running tests that require a Kubernetes API server
+# SETUP_ENVTEST_VER is the version of setup-envtest to use, matching the version in hack/tools/go.mod
+KUBEBUILDER_ENVTEST_KUBERNETES_VERSION ?= 1.34.0


4.22 is 1.35, pretty sure we have 1.35 binaries published that you can use

I did take 1.34 because that is what hypershift is currently built with. If I should follow OCP k8s version instead I will look into this as part of the above effort to make the setup parametric.

enxebre · 2026-03-23T10:29:18Z

+	crdObjects := installassets.CustomResourceDefinitions(
+		func(path string, crd *apiextensionsv1.CustomResourceDefinition) bool {
+			if !strings.Contains(path, "hypershift-operator/") {
+				return false


this excludes all the API CRDs which we would want to include as well

jparrill · 2026-03-23T15:24:09Z

/retitle CNTRLPLANE-3037,OCPBUGS-78837: Introduce envtest for integration tests

openshift-ci-robot · 2026-03-23T15:24:28Z

@clebs: This pull request references CNTRLPLANE-3037 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set.

This pull request references Jira Issue OCPBUGS-78837, which is invalid:

expected the bug to target the "4.22.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

What this PR does / why we need it:

This PR introduces envtest and replicates the existing TestOnCreateAPIUX e2e test (but not replacing it yet).

Shifting part of the tests away from e2e into integration tests that do not require a full environment will help reduce the burden on our CI.

Checklist:

Subject and description added to both, commit and PR.

Relevant issues have been referenced.

This change includes docs.

This change includes unit tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

jparrill · 2026-03-23T15:25:42Z

The jira above was created to track this issue OCPBUGS-78837. Discussed on slack that this would be detected using this testEnv.

enxebre · 2026-03-23T19:35:29Z

superseded by #8034

openshift-ci · 2026-03-26T22:32:27Z

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-ci · 2026-03-30T15:57:42Z

@clebs: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/security	`e622b97`	link	true	`/test security`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

clebs · 2026-03-31T14:02:33Z

/close

An alternative PR merged #8089

openshift-ci · 2026-03-31T14:05:33Z

@clebs: Closed this PR.

Details

In response to this:

/close

An alternative PR merged #8089

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Mar 19, 2026

openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 19, 2026

openshift-ci Bot added the do-not-merge/needs-area label Mar 19, 2026

openshift-ci Bot added area/ci-tooling Indicates the PR includes changes for CI or tooling area/testing Indicates the PR includes changes for e2e testing and removed do-not-merge/needs-area labels Mar 19, 2026

clebs force-pushed the envtest-poc branch from 402b9f6 to 48bab4d Compare March 19, 2026 16:46

clebs force-pushed the envtest-poc branch from 48bab4d to eaaab91 Compare March 19, 2026 18:58

enxebre changed the title ~~NO-JIRA: Introduce envtest for integration tests~~ CNTRLPLANE-3037: Introduce envtest for integration tests Mar 20, 2026

enxebre reviewed Mar 20, 2026

View reviewed changes

openshift-ci Bot assigned JoelSpeed Mar 20, 2026

openshift-ci Bot added the area/api Indicates the PR includes changes for the API label Mar 20, 2026

enxebre mentioned this pull request Mar 20, 2026

CNTRLPLANE-2939: Coordinate CRD lifecycle with Cluster CAPI Operator #7996

Merged

enxebre mentioned this pull request Mar 20, 2026

CNTRLPLANE-2049: fix(api): audit and improve GCP API validation rules #8003

Merged

4 tasks

clebs added 2 commits March 20, 2026 17:51

test(unit): Introduce envtest

e8579e8

- Add an envtest setup goal on the Makefile - Add a new unit test using envtest to cover the same as the create cluster e2e test. Signed-off-by: Borja Clemente <bclement@redhat.com>

build(deps): vendor dependencies needed for envtest

e622b97

Introducing envtest requires adding new dependencies, which are being vendored in a separate commit to ease review. Signed-off-by: Borja Clemente <bclement@redhat.com>

clebs force-pushed the envtest-poc branch from eaaab91 to e622b97 Compare March 20, 2026 16:51

JoelSpeed reviewed Mar 21, 2026

View reviewed changes

enxebre reviewed Mar 23, 2026

View reviewed changes

enxebre mentioned this pull request Mar 23, 2026

CNTRLPLANE-3037: Introduce envtest for integration tests #8034

Closed

4 tasks

openshift-ci Bot changed the title ~~CNTRLPLANE-3037: Introduce envtest for integration tests~~ CNTRLPLANE-3037,OCPBUGS-78837: Introduce envtest for integration tests Mar 23, 2026

openshift-ci-robot added the jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. label Mar 23, 2026

openshift-ci Bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 26, 2026

openshift-ci Bot closed this Mar 31, 2026

clebs deleted the envtest-poc branch August 4, 2026 12:28

Uh oh!

Conversation

clebs commented Mar 19, 2026

What this PR does / why we need it:

Checklist:

Uh oh!

openshift-ci-robot commented Mar 19, 2026

Uh oh!

openshift-ci-robot commented Mar 19, 2026

What this PR does / why we need it:

Checklist:

Uh oh!

openshift-ci Bot commented Mar 19, 2026

Uh oh!

coderabbitai Bot commented Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

openshift-ci Bot commented Mar 19, 2026

Uh oh!

clebs commented Mar 19, 2026

Uh oh!

clebs commented Mar 19, 2026

Uh oh!

openshift-ci-robot commented Mar 20, 2026 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does / why we need it:

Checklist:

Uh oh!

enxebre Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

enxebre Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

enxebre commented Mar 20, 2026

Uh oh!

JoelSpeed commented Mar 20, 2026

Uh oh!

enxebre commented Mar 20, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jparrill commented Mar 23, 2026 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Mar 23, 2026 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does / why we need it:

Checklist:

Uh oh!

jparrill commented Mar 23, 2026

Uh oh!

enxebre commented Mar 23, 2026

Uh oh!

openshift-ci Bot commented Mar 26, 2026

Uh oh!

openshift-ci Bot commented Mar 30, 2026

Uh oh!

clebs commented Mar 31, 2026

Uh oh!

openshift-ci Bot commented Mar 31, 2026

coderabbitai Bot commented Mar 19, 2026 •

edited

Loading

openshift-ci-robot commented Mar 20, 2026 •

edited by openshift-ci Bot

Loading

enxebre Mar 20, 2026 •

edited

Loading

enxebre Mar 20, 2026 •

edited

Loading

jparrill commented Mar 23, 2026 •

edited by openshift-ci Bot

Loading

openshift-ci-robot commented Mar 23, 2026 •

edited by openshift-ci Bot

Loading