fix(ci): align hermetic build with must-gather improvements by Fortune-Ndlovu · Pull Request #3668 · redhat-developer/rhdh-operator

Fortune-Ndlovu · 2026-09-30T14:09:35Z

Summary

Follow-up to rhdh-must-gather#218: apply the same hermetic CI hygiene to the operator.

Single Hermeto pin in the Makefile (Renovate + shared by CI and make hermetic-build)
Cache key includes the Hermeto image tag
chmod on the prefetched cache after inject (local script + CI) so the go-builder can stay non-root
Remove USER 0 from the builder stage; set GOCACHE=/tmp/go-build-cache

Test plan

make hermetic-build IMG=localhost/rhdh-operator:hermetic-test
podman run --rm localhost/rhdh-operator:hermetic-test --help
PR Build (hermetic) green on GitHub Actions

rhdh-qodo-merge · 2026-09-30T14:10:16Z

PR Summary by Qodo

Align hermetic CI and local builds around a shared Hermeto pin

🐞 Bug fix ⚙️ Configuration changes 🕐 20-40 Minutes

AI Description

• Share one Renovate-managed Hermeto image pin across CI and local builds.
• Invalidate CI dependency caches when the Hermeto tag changes.
• Make prefetched caches writable so the Go builder can run as non-root.

Diagram

graph TD
  R["Renovate"] --> M["Makefile pin"] --> C["CI build action"] --> H[("Hermeto cache")] --> B["Non-root builder"] --> I["Operator image"]
  M --> L["Local build script"] --> H

High-Level Assessment

Keep the shared Makefile pin: it avoids separate CI and script versions while remaining Renovate-managed. Duplicating pins would reintroduce drift, and a separate configuration file would add another layer for a single value.

Files changed (6) +50 / -9

Bug fix (3) +33 / -7

action.yamlUse the shared pin and prepare a writable CI cache +14/-4

Use the shared pin and prepare a writable CI cache
• Reads the Hermeto image from the Makefile and includes its tag in the dependency-cache key. Makes the cache writable after injection so the non-root builder can use it.
.github/actions/docker-build/action.yaml

DockerfileRun the Go builder as the default non-root user +3/-2

Run the Go builder as the default non-root user
• Removes the builder's root-user override and places the Go build cache under /tmp.
Dockerfile

local-hermeto-build.shRead the shared pin and open cache permissions locally +16/-1

Read the shared pin and open cache permissions locally
• Accepts an image override or reads the component Makefile, failing if neither supplies a value. Makes the prefetched cache writable after injection for non-root image builds.
scripts/local-hermeto-build.sh

Documentation (1) +2 / -0

developer.mdDocument the shared Hermeto image pin +2/-0

Document the shared Hermeto image pin
• Explains that CI and local hermetic builds use the image value defined in the Makefile.
docs/developer.md

Other (2) +15 / -2

renovate.jsonTrack the Makefile Hermeto pin with Renovate +12/-1

Track the Makefile Hermeto pin with Renovate
• Adds a regex manager that discovers the Hermeto Docker image version in the Makefile.
.github/renovate.json

MakefileDefine and pass the shared Hermeto image pin +3/-1

Define and pass the shared Hermeto image pin
• Defines the Renovate-managed image value and passes it to the local hermetic-build script.
Makefile

rhdh-qodo-merge · 2026-09-30T14:11:58Z

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

1. ~~Some local cache builds stop at chmod~~ ✓ Resolved

Description

build_cache runs an unguarded host-side chmod -R on files populated by Podman containers,
without first ensuring the invoking user owns them. If those files are owned by another host UID,
chmod fails under set -e and the image build never starts; the CI path explicitly repairs
ownership before its equivalent chmod.

Code

scripts/local-hermeto-build.sh[174]
+  chmod -R a+rwX "${local_cache_dir}"

Relevance

●●● Strong
Prior reviewers accepted deterministic local-script reliability fixes; missing ownership repair can
terminate builds under set -e.
PR-#3359

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
Podman commands populate the mounted cache before the new host-side chmod, and set -e makes a
permission failure fatal. Unlike the local script, CI performs a privileged recursive chown before
chmod.
scripts/local-hermeto-build.sh[16-16]
scripts/local-hermeto-build.sh[136-175]
scripts/local-hermeto-build.sh[301-310]
.github/actions/docker-build/action.yaml[139-145]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
A host-side chmod terminates local builds when Podman-created cache files belong to a different host UID.
## Fix Focus Areas
- scripts/local-hermeto-build.sh[136-175]
- .github/actions/docker-build/action.yaml[139-145]
## Recommended Fix
Normalize cache ownership using an appropriate Podman user-namespace operation before changing permissions, or perform the permission change as a user that owns the files. Verify the script works with rootless Podman cache files owned by subordinate UIDs.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

2. ~~Local users can poison build dependencies~~ ✓ Resolved

Description

build_cache applies chmod -R a+rwX to the entire prefetched cache, allowing other local users to
alter its contents. When the cache path is accessible to another user, their changes to the
generated environment file or cached dependencies are consumed by the subsequent offline image
build.

Code

scripts/local-hermeto-build.sh[174]
+  chmod -R a+rwX "${local_cache_dir}"

Relevance

●● Moderate
Intent explicitly matches CI hygiene, but broad writable permissions create a credible local
cache-integrity risk.
PR-#3359

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The new chmod grants write access to all users; the script then mounts that same directory at
/cachi2, and its Dockerfile transformation sources the environment file from that mount during
build commands.
scripts/local-hermeto-build.sh[169-174]
scripts/local-hermeto-build.sh[118-124]
scripts/local-hermeto-build.sh[198-214]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Recursive all-user write permission lets other local users alter dependencies used by the offline image build.
## Fix Focus Areas
- scripts/local-hermeto-build.sh[169-175]
- scripts/local-hermeto-build.sh[198-214]
## Recommended Fix
Give the build container access through a restricted group or an appropriate user-namespace ownership mapping instead of making the entire cache writable by everyone.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Context sources

✅ Compliance rules (platform): 18 rules

✅ Cross-repo context — repo relationships

Explored: repo: redhat-developer/rhdh-must-gather (sha: ad4beb40) — View relationship

Review mode: ⚖️ Balanced: This changes hermetic CI/build behavior across Makefile, shell, Dockerfile, caching, permissions, and Renovate configuration, creating meaningful cross-path correctness and security implications without being dense enough to require extended review.

Tip of the day

💡 Did you know, you can show, collapse, or hide each part of a finding: code, evidence, and all

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Pin Hermeto once in the Makefile for CI and local builds, include the Hermeto tag in the deps cache key, chmod the prefetch cache for non-root go-builder, and drop USER 0 from the builder stage. Signed-off-by: Fortune-Ndlovu <fndlovu@redhat.com>

rhdh-qodo-merge · 2026-09-30T14:12:10Z

Important

The /generate_labels command by Qodo is sunsetting on the 1st of October 2026 and will no longer be available. We recommend switching to the latest Qodo review capabilities. Learn more

Chown the Podman-populated cache before chmod (podman unshare, like CI), use /tmp/hermeto-cache/operator for local builds, and restrict the cache parent directory to the invoking user. Signed-off-by: Fortune-Ndlovu <fndlovu@redhat.com>

github-actions · 2026-09-30T14:17:14Z

✅ PR images built and pushed successfully!

Images are available for testing (expires in 7 days):

Image	Full tag	PR tag
Operator	`quay.io/rhdh-community/operator:2.1.0-pr-3668-c83d33b`	`quay.io/rhdh-community/operator:2.1.0-pr-3668`
Bundle	`quay.io/rhdh-community/operator-bundle:2.1.0-pr-3668-c83d33b`	`quay.io/rhdh-community/operator-bundle:2.1.0-pr-3668`
Catalog	`quay.io/rhdh-community/operator-catalog:2.1.0-pr-3668-c83d33b`	`quay.io/rhdh-community/operator-catalog:2.1.0-pr-3668`

github-actions · 2026-09-30T14:21:24Z

✅ PR images built and pushed successfully!

Images are available for testing (expires in 7 days):

Image	Full tag	PR tag
Operator	`quay.io/rhdh-community/operator:2.1.0-pr-3668-fe4df57`	`quay.io/rhdh-community/operator:2.1.0-pr-3668`
Bundle	`quay.io/rhdh-community/operator-bundle:2.1.0-pr-3668-fe4df57`	`quay.io/rhdh-community/operator-bundle:2.1.0-pr-3668`
Catalog	`quay.io/rhdh-community/operator-catalog:2.1.0-pr-3668-fe4df57`	`quay.io/rhdh-community/operator-catalog:2.1.0-pr-3668`

Map go-builder and rpm-builder Podman UIDs with setfacl (subuid math plus a root probe) so local offline builds work without world-writable cache files. Signed-off-by: Fortune-Ndlovu <fndlovu@redhat.com>

github-actions · 2026-09-30T14:28:21Z

✅ PR images built and pushed successfully!

Images are available for testing (expires in 7 days):

Image	Full tag	PR tag
Operator	`quay.io/rhdh-community/operator:2.1.0-pr-3668-f441c05`	`quay.io/rhdh-community/operator:2.1.0-pr-3668`
Bundle	`quay.io/rhdh-community/operator-bundle:2.1.0-pr-3668-f441c05`	`quay.io/rhdh-community/operator-bundle:2.1.0-pr-3668`
Catalog	`quay.io/rhdh-community/operator-catalog:2.1.0-pr-3668-f441c05`	`quay.io/rhdh-community/operator-catalog:2.1.0-pr-3668`

Relocate local Hermeto helper to hack/local-hermeto-build.sh, add CONTAINERFILE and per-image cache dirs under /tmp/hermeto-cache/, and parse multi-stage Dockerfiles for ACL UID mapping. Add dp-installer-hermetic-build for plugin-installer. Signed-off-by: Fortune-Ndlovu <fndlovu@redhat.com>

github-actions · 2026-10-01T14:52:17Z

✅ PR images built and pushed successfully!

Images are available for testing (expires in 7 days):

Image	Full tag	PR tag
Operator	`quay.io/rhdh-community/operator:2.2.0-pr-3668-f235d26`	`quay.io/rhdh-community/operator:2.2.0-pr-3668`
Bundle	`quay.io/rhdh-community/operator-bundle:2.2.0-pr-3668-f235d26`	`quay.io/rhdh-community/operator-bundle:2.2.0-pr-3668`
Catalog	`quay.io/rhdh-community/operator-catalog:2.2.0-pr-3668-f235d26`	`quay.io/rhdh-community/operator-catalog:2.2.0-pr-3668`

Use podman unshare chown plus chmod -R a+rwX on the local Hermeto cache, same as rhdh-must-gather redhat-developer#218 and CI. Removes the acl package requirement. Signed-off-by: Fortune-Ndlovu <fndlovu@redhat.com>

sonarqubecloud · 2026-10-02T12:08:26Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

github-actions · 2026-10-02T12:12:34Z

✅ PR images built and pushed successfully!

Images are available for testing (expires in 7 days):

Image	Full tag	PR tag
Operator	`quay.io/rhdh-community/operator:2.2.0-pr-3668-3a91f09`	`quay.io/rhdh-community/operator:2.2.0-pr-3668`
Bundle	`quay.io/rhdh-community/operator-bundle:2.2.0-pr-3668-3a91f09`	`quay.io/rhdh-community/operator-bundle:2.2.0-pr-3668`
Catalog	`quay.io/rhdh-community/operator-catalog:2.2.0-pr-3668-3a91f09`	`quay.io/rhdh-community/operator-catalog:2.2.0-pr-3668`

codecov · 2026-10-02T12:13:51Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 60.43%. Comparing base (0379ecd) to head (3a91f09).
⚠️ Report is 12 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3668      +/-   ##
==========================================
+ Coverage   60.09%   60.43%   +0.33%     
==========================================
  Files          51       52       +1     
  Lines        3614     3672      +58     
==========================================
+ Hits         2172     2219      +47     
- Misses       1247     1257      +10     
- Partials      195      196       +1

Flag	Coverage Δ
nightly	`?`
unittests	`60.43% <ø> (+0.33%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 7 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Fortune-Ndlovu requested a review from a team as a code owner September 30, 2026 14:09

openshift-ci Bot requested review from gazarenkov and rm3l September 30, 2026 14:09

rhdh-qodo-merge Bot added documentation Improvements or additions to documentation enhancement New feature or request Bug fix labels Sep 30, 2026

Fortune-Ndlovu force-pushed the improve-hermetic-build-cache-and-non-root branch from 3a8a3ae to c83d33b Compare September 30, 2026 14:12

fix(ci): grant hermeto cache via ACLs instead of a+rwX

f441c05

Map go-builder and rpm-builder Podman UIDs with setfacl (subuid math plus a root probe) so local offline builds work without world-writable cache files. Signed-off-by: Fortune-Ndlovu <fndlovu@redhat.com>

rm3l reviewed Oct 1, 2026

View reviewed changes

Comment thread hack/local-hermeto-build.sh Outdated

fix(ci): match must-gather cache permissions, drop setfacl

3a91f09

Use podman unshare chown plus chmod -R a+rwX on the local Hermeto cache, same as rhdh-must-gather redhat-developer#218 and CI. Removes the acl package requirement. Signed-off-by: Fortune-Ndlovu <fndlovu@redhat.com>

gazarenkov approved these changes Oct 2, 2026

View reviewed changes

openshift-ci Bot assigned gazarenkov Oct 2, 2026

openshift-ci Bot added the lgtm label Oct 2, 2026

openshift-merge-bot Bot merged commit 088febb into redhat-developer:main Oct 2, 2026
14 checks passed

Navigation Menu

fix(ci): align hermetic build with must-gather improvements - #3668

fix(ci): align hermetic build with must-gather improvements#3668
openshift-merge-bot[bot] merged 5 commits into
redhat-developer:mainfrom
Fortune-Ndlovu:improve-hermetic-build-cache-and-non-root

Fortune-Ndlovu commented Sep 30, 2026 •

edited

Loading

Uh oh!

rhdh-qodo-merge Bot commented Sep 30, 2026

Uh oh!

rhdh-qodo-merge Bot commented Sep 30, 2026 •

edited

Loading

Uh oh!

rhdh-qodo-merge Bot commented Sep 30, 2026

Uh oh!

github-actions Bot commented Sep 30, 2026

Uh oh!

github-actions Bot commented Sep 30, 2026

Uh oh!

github-actions Bot commented Sep 30, 2026

Uh oh!

github-actions Bot commented Oct 1, 2026

Uh oh!

Uh oh!

sonarqubecloud Bot commented Oct 2, 2026

Uh oh!

github-actions Bot commented Oct 2, 2026

Uh oh!

codecov Bot commented Oct 2, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Fortune-Ndlovu commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

rhdh-qodo-merge Bot commented Sep 30, 2026

PR Summary by Qodo

Uh oh!

rhdh-qodo-merge Bot commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review by Qodo

Uh oh!

rhdh-qodo-merge Bot commented Sep 30, 2026

Uh oh!

github-actions Bot commented Sep 30, 2026

Uh oh!

github-actions Bot commented Sep 30, 2026

Uh oh!

github-actions Bot commented Sep 30, 2026

Uh oh!

github-actions Bot commented Oct 1, 2026

Uh oh!

Uh oh!

sonarqubecloud Bot commented Oct 2, 2026

Quality Gate passed

Uh oh!

github-actions Bot commented Oct 2, 2026

Uh oh!

codecov Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Fortune-Ndlovu commented Sep 30, 2026 •

edited

Loading

rhdh-qodo-merge Bot commented Sep 30, 2026 •

edited

Loading

codecov Bot commented Oct 2, 2026 •

edited

Loading