fix: add CPO and CVO Quay exceptions to image registry VAP for all environments by avollmer-redhat · Pull Request #7242 · Azure/ARO-HCP

Anthony Vollmer (avollmer-redhat) · 2026-09-29T13:03:33Z

Why

HyperShift does not rewrite two categories of Quay image references through ACR pull-through cache before writing them into pod specs, causing ValidatingAdmissionPolicy violations:

CPO (Control Plane Operator): GetControlPlaneOperatorImage() returns raw quay.io/redhat-user-workloads/crt-redhat-acm-tenant/control-plane-operator-* URLs from the override table. The versioned repos (e.g. control-plane-operator-4-19, -4-20) require a namespace-level prefix exception. Upstream fix: OCPBUGS-128660.
CVO prepare-payload init container: The OCP release image quay.io/openshift-release-dev/ocp-release is resolved with ICSP/IDMS mirrors, but the pod spec still carries the original quay.io prefix. Upstream fix: OCPBUGS-128661.

What

Add quay.io/redhat-user-workloads/crt-redhat-acm-tenant to global defaults.imageRegistryPolicy.extraAllowedRegistries — namespace-level prefix to match the ACR pull-through cache rule scope (global-acr.bicep) and the HyperShift registryOverrides mapping, covering all current and future versioned CPO repos
Add quay.io/openshift-release-dev/ocp-release to the same list
Re-materialize configs and Helm fixtures

Rollback plan

These exceptions should be rolled back once the upstream HyperShift fixes (OCPBUGS-128660, OCPBUGS-128661) merge and propagate to the OCP versions we deploy.

Checklist

Config change with make materialize re-run
Rendered configs committed
Helm fixtures updated
OCPBUGS filed for upstream tracking

openshift-merge-bot · 2026-09-29T13:03:36Z

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

openshift-ci · 2026-09-29T13:03:45Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

Copilot

Copilot review overview

🟡 Changes recommended

The shared configuration unnecessarily permits direct Quay CPO images on service clusters.

Review effort: Balanced
Findings: 1

Open (1)

Keep Quay CPO image allowlist restricted to management clusters · New

What changed in this PR

Restores the dev-cloud CPO Quay exception needed for HyperShift CPO overrides.

Changes:

Adds the CPO image prefix to dev allowlists.
Documents the workaround and long-term fix.
Regenerates six rendered configs and Helm fixtures.

File	Description
`config/config.yaml`	Adds and documents the exception.
`config/rendered/dev/cspr/westus3.yaml`	Renders CSPR configuration.
`config/rendered/dev/ci00/centralus.yaml`	Renders CI00 configuration.
`config/rendered/dev/ci01/centralus.yaml`	Renders CI01 configuration.
`config/rendered/dev/dev/westus3.yaml`	Renders dev configuration.
`config/rendered/dev/perf/westus3.yaml`	Renders performance configuration.
`config/rendered/dev/pers/westus3.yaml`	Renders personal-dev configuration.
`dev-infrastructure/zz_fixture_TestHelmTemplate_dev_westus3_mgmt_1_image_registry_policy.yaml`	Updates the management-cluster fixture.
`dev-infrastructure/zz_fixture_TestHelmTemplate_dev_westus3_svc_1_image_registry_policy.yaml`	Updates the service-cluster fixture.

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+      # path before setting the annotation (tracked in AROSLSRE-861).
      imageRegistryPolicy:
-        extraAllowedRegistries: docker.io/grafana,docker.io/library/postgres
+        extraAllowedRegistries: docker.io/grafana,docker.io/library/postgres,quay.io/redhat-user-workloads/crt-redhat-acm-tenant/control-plane-operator


openshift-merge-bot · 2026-09-29T15:45:52Z

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-parallel

Manyanda Chitimbo (machi1990) · 2026-09-29T15:53:24Z

/lgtm

Manyanda Chitimbo (machi1990)

/lgtm cancel

Manyanda Chitimbo (machi1990) · 2026-09-29T15:57:18Z

+      # path before setting the annotation (tracked in AROSLSRE-861).
      imageRegistryPolicy:
-        extraAllowedRegistries: docker.io/grafana,docker.io/library/postgres
+        extraAllowedRegistries: docker.io/grafana,docker.io/library/postgres,quay.io/redhat-user-workloads/crt-redhat-acm-tenant/control-plane-operator


Anthony Vollmer (@avollmer-redhat) are you absolute sure that we'd not also want this exception to be in all envs (prod included) until the fix lands in hypershift?

Yes — the exceptions are in global defaults now, so INT/STG/PROD inherit them. The dev overlay still needs to repeat them because extraAllowedRegistries is a scalar string (not a list), so the dev cloud override completely replaces the defaults value. If we didn't repeat them in the dev overlay, dev would lose the quay.io exceptions and only have the docker.io ones.

Copilot

Copilot review overview

🟡 Changes recommended

The CPO entry does not match suffixed repositories, and restricted operational details require removal.

Review effort: Balanced
Findings: 2 · 1

Open (3)

CPO wildcard repository is rejected by allowlist matcher · New
Keep Quay CPO image allowlist restricted to management clusters
Remove disclosed internal compliance-policy implementation details · New

Manyanda Chitimbo (machi1990) · 2026-09-29T16:17:13Z

+      # Image Registry Policy — dev-only extras on top of the base quay.io
+      # exceptions (which are inherited from defaults.imageRegistryPolicy).
      imageRegistryPolicy:
-        extraAllowedRegistries: docker.io/grafana,docker.io/library/postgres
+        extraAllowedRegistries: docker.io/grafana,docker.io/library/postgres,quay.io/redhat-user-workloads/crt-redhat-acm-tenant/control-plane-operator,quay.io/openshift-release-dev/ocp-release


With the change above, this isn't needed?

Still needed — the dev cloud override of extraAllowedRegistries completely replaces the global defaults value (scalar string merge semantics, not append). Without repeating the quay.io entries here, dev environments would only have the docker.io prefixes.

openshift-merge-bot · 2026-09-29T16:33:45Z

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-parallel

Anthony Vollmer (avollmer-redhat)

Addressing the review feedback — pushing an updated commit shortly.

Anthony Vollmer (avollmer-redhat) · 2026-09-29T17:09:28Z

+  # fixes merge and propagate.
  imageRegistryPolicy:
-    extraAllowedRegistries: ""
+    extraAllowedRegistries: quay.io/redhat-user-workloads/crt-redhat-acm-tenant/control-plane-operator,quay.io/openshift-release-dev/ocp-release


Good catch — the current prefix control-plane-operator doesn't match the versioned repos (control-plane-operator-4-19, control-plane-operator-4-20) because the CEL only allows /, :, @ as separators, not -.

We're widening to the namespace-level prefix quay.io/redhat-user-workloads/crt-redhat-acm-tenant to match the ACR pull-through cache rule scope (global-acr.bicep line 90) and the HyperShift registryOverrides mapping (hypershiftoperator/values.yaml line 7), which both already operate at this level. This also avoids needing to enumerate and maintain per-OCP-version repos as new versions get CPO overrides.

Anthony Vollmer (avollmer-redhat) · 2026-09-29T17:09:28Z

+  # references through ACR pull-through cache before writing them into pod
+  # specs, so the VAP must allow them until upstream fixes land.
+  #   - control-plane-operator: GetControlPlaneOperatorImage() returns raw
+  #     quay.io URLs from the CPO override table (OCPBUGS-51657).


Agreed — trimming the comments down to just reference the upstream bug IDs without describing internal policy behavior.

Copilot

Copilot review overview

🟡 Changes recommended

The exceptions unnecessarily weaken service-cluster admission policy despite only addressing management-cluster workloads.

Review effort: Balanced
Findings: 1 · 1

Open (2)

Scope Quay exceptions to management-cluster policy only · New
Keep Quay CPO image allowlist restricted to management clusters

Resolved since last review (2)

CPO wildcard repository is rejected by allowlist matcher
Remove disclosed internal compliance-policy implementation details

…vironments HyperShift does not rewrite certain Quay image references through ACR pull-through cache before writing them into pod specs. Add temporary VAP exceptions until upstream fixes land: - crt-redhat-acm-tenant: CPO override images use versioned repos (e.g. control-plane-operator-4-19, -4-20). The namespace-level prefix matches the ACR pull-through cache rule and HyperShift registryOverrides scope. (OCPBUGS-128660) - ocp-release: CVO prepare-payload image (OCPBUGS-128661) AROSLSRE-861

Copilot

Copilot review overview

🟢 Approval recommended

The scoped configuration change correctly matches both affected image prefixes and its generated artifacts are consistent.

Review effort: Balanced
Findings: 1

Open (1)

Keep Quay CPO image allowlist restricted to management clusters

Resolved since last review (1)

Scope Quay exceptions to management-cluster policy only

openshift-merge-bot · 2026-09-29T19:18:36Z

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-parallel

avollmer-ms · 2026-09-30T15:35:45Z

/retest-required

Anthony Vollmer (avollmer-redhat) · 2026-09-30T15:55:44Z

Test Failure Analysis

CodeQL Analysis Failure (Exit Code 143)

The CodeQL analysis is failing with exit code 143 (killed/OOM). This is a resource issue in the CI environment, not a problem with the VAP fix code itself.

Exit code 143 typically means the process was killed due to:

Out of memory (OOM)
Timeout
Resource limits exceeded

The Go build step for CodeQL ran for 6m 19s before being killed. This is a CI infrastructure issue, not a code quality problem.

Prometheus Rules Test Failure

The TestPrometheusRules tests are failing with:

testing rules failed error running promtool exec: "promtool": executable file not found in $PATH

This is a pre-existing infrastructure issue unrelated to the VAP fix. The promtool binary is not available in the test environment. This failure exists on main branch as well and should be fixed independently.

Summary

The VAP fix itself is correct and ready:

✅ Config changes are minimal and correct (adding Quay registries to allowlist)
✅ All relevant Helm test fixtures properly updated
✅ Unit tests pass
❌ CodeQL: CI infrastructure issue (OOM/timeout)
❌ Prometheus: Pre-existing environment issue (promtool missing)

Both test failures are environmental issues unrelated to the code change.

Himanshu Bhushan (hbhushan3) · 2026-09-30T20:52:59Z

/lgtm

openshift-ci · 2026-09-30T20:53:09Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: avollmer-redhat, hbhushan3

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~config/OWNERS~~ [avollmer-redhat,hbhushan3]
~~dev-infrastructure/OWNERS~~ [avollmer-redhat,hbhushan3]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

avollmer-ms · 2026-09-30T21:43:54Z

/retest-required

openshift-merge-bot · 2026-10-01T00:52:35Z

/retest-required

Remaining retests: 0 against base HEAD 21717c1 and 2 for PR HEAD 136c196 in total

Steven Clarkson (sclarkso) · 2026-10-01T03:51:17Z

/retest-required

Copilot AI balanced review requested due to automatic review settings September 29, 2026 13:03

openshift-ci Bot added do-not-merge/work-in-progress area/infrastructure Deployment/pipeline infrastructure config approved labels Sep 29, 2026

Copilot started reviewing on behalf of Anthony Vollmer (avollmer-redhat) September 29, 2026 13:04 View session

Copilot AI reviewed Sep 29, 2026

View reviewed changes

Anthony Vollmer (avollmer-redhat) marked this pull request as ready for review September 29, 2026 15:00

openshift-ci Bot removed the do-not-merge/work-in-progress label Sep 29, 2026

openshift-ci Bot requested review from Gerd Oberlechner (geoberle) and Jan-Hendrik Boll (janboll) September 29, 2026 15:01

openshift-ci Bot assigned Manyanda Chitimbo (machi1990) Sep 29, 2026

openshift-ci Bot added the lgtm label Sep 29, 2026

Manyanda Chitimbo (machi1990) reviewed Sep 29, 2026

View reviewed changes

openshift-ci Bot removed the lgtm label Sep 29, 2026

Manyanda Chitimbo (machi1990) reviewed Sep 29, 2026

View reviewed changes

Anthony Vollmer (avollmer-redhat) force-pushed the fix/aroslsre-861-restore-cpo-vap-exception-dev branch from 1d21c32 to db3d08e Compare September 29, 2026 16:07

Copilot AI review requested due to automatic review settings September 29, 2026 16:07

Anthony Vollmer (avollmer-redhat) changed the title ~~fix: restore CPO Quay exception in dev-cloud image registry VAP~~ fix: add CPO and CVO Quay exceptions to image registry VAP for all environments Sep 29, 2026

Copilot started reviewing on behalf of Anthony Vollmer (avollmer-redhat) September 29, 2026 16:08 View session

Copilot AI reviewed Sep 29, 2026

View reviewed changes

Comment thread config/config.yaml Outdated

Comment thread config/config.yaml Outdated

Manyanda Chitimbo (machi1990) reviewed Sep 29, 2026

View reviewed changes

Anthony Vollmer (avollmer-redhat) commented Sep 29, 2026

View reviewed changes

Copilot AI balanced review requested due to automatic review settings September 29, 2026 17:20

Anthony Vollmer (avollmer-redhat) force-pushed the fix/aroslsre-861-restore-cpo-vap-exception-dev branch from db3d08e to 261c345 Compare September 29, 2026 17:20

Copilot started reviewing on behalf of Anthony Vollmer (avollmer-redhat) September 29, 2026 17:21 View session

Copilot AI reviewed Sep 29, 2026

View reviewed changes

Comment thread dev-infrastructure/zz_fixture_TestHelmTemplate_dev_westus3_svc_1_image_registry_policy.yaml

Copilot AI balanced review requested due to automatic review settings September 29, 2026 17:55

Anthony Vollmer (avollmer-redhat) force-pushed the fix/aroslsre-861-restore-cpo-vap-exception-dev branch from 261c345 to 136c196 Compare September 29, 2026 17:55

Copilot started reviewing on behalf of Anthony Vollmer (avollmer-redhat) September 29, 2026 17:56 View session

Copilot AI reviewed Sep 29, 2026

View reviewed changes

openshift-ci Bot assigned Himanshu Bhushan (hbhushan3) Sep 30, 2026

openshift-ci Bot added the lgtm label Sep 30, 2026

openshift-merge-bot Bot merged commit bb2cbe0 into Azure:main Oct 1, 2026
16 of 18 checks passed

Conversation

Anthony Vollmer (avollmer-redhat) commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

What

Rollback plan

Related

Checklist

Uh oh!

openshift-merge-bot Bot commented Sep 29, 2026

Uh oh!

openshift-ci Bot commented Sep 29, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

openshift-merge-bot Bot commented Sep 29, 2026

Uh oh!

Manyanda Chitimbo (machi1990) commented Sep 29, 2026

Uh oh!

Manyanda Chitimbo (machi1990) left a comment

Choose a reason for hiding this comment

Uh oh!

Manyanda Chitimbo (machi1990) Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Anthony Vollmer (avollmer-redhat) Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Manyanda Chitimbo (machi1990) Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

Anthony Vollmer (avollmer-redhat) Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

openshift-merge-bot Bot commented Sep 29, 2026

Uh oh!

Anthony Vollmer (avollmer-redhat) left a comment

Choose a reason for hiding this comment

Uh oh!

Anthony Vollmer (avollmer-redhat) Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

Anthony Vollmer (avollmer-redhat) Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟢 Approval recommended

Uh oh!

openshift-merge-bot Bot commented Sep 29, 2026

Uh oh!

avollmer-ms commented Sep 30, 2026

Uh oh!

Anthony Vollmer (avollmer-redhat) commented Sep 30, 2026

Test Failure Analysis

CodeQL Analysis Failure (Exit Code 143)

Prometheus Rules Test Failure

Summary

Uh oh!

Himanshu Bhushan (hbhushan3) commented Sep 30, 2026

Uh oh!

openshift-ci Bot commented Sep 30, 2026

Anthony Vollmer (avollmer-redhat) commented Sep 29, 2026 •

edited

Loading

Manyanda Chitimbo (machi1990) Sep 29, 2026 •

edited

Loading