Repository navigation
Conversation
…planes, e2-standard-4 workers Final production GCP sizing scheme, based on two rehearsal campaigns (openshift#82754, openshift#82856): - Control planes: change the shared ipi-conf-gcp default from e2-custom-6-16384 (6 vCPU / 16GB) to e2-custom-8-16384 (8 vCPU / 16GB). The prior default carried a code comment noting it was a "temporary test to see if this helps the consistent high CPU alerts and random test failures" - confirmed via a 12-run rehearsal campaign across 3 alternative instance types (openshift#82754) and a 30-day production survey across ~20 GCP 5.0 jobs, both showing e2-custom-8-16384 matches or beats e2-standard-8 on etcd fsync/commit latency and CPU saturation at ~13% lower cost. vCPU count (not memory ratio or CPU host platform) was the dominant driver: n2-standard-4 (4 vCPU) performed worse than the 6 vCPU baseline, every 8-vCPU option performed better. - Serial control planes: pinned back down to e2-custom-6-16384 via an explicit CONTROL_PLANE_NODE_TYPE override on every serial GCP job (51 jobs across 33 config files, using workflows openshift-e2e-gcp-serial{,-runc,-cgroupsv1}, openshift-e2e-gcp-ovn-serial, openshift-e2e-gcp-sdn-serial, openshift-e2e-gcp-techpreview-serial). Serial jobs run a non-parallel test suite and already show lower control-plane CPU utilization than non-serial jobs, and a 3-round study (openshift#82856) found downsizing them further to e2-standard-4 (4 vCPU) causes a clear, reproducible etcd disk-latency regression (fsync p99 elevated 29-113% above baseline in every round with data, including one round with real etcd overload errors: "etcdserver: too many requests", leader elections). A handful of old (nightly-4.6-4.16) jobs have "serial" in their as: name but run the parallel workflow (openshift-e2e-gcp/openshift-e2e-gcp-sdn) - left alone, since serial-ness is determined by workflow, not job name. - Workers: change the shared COMPUTE_NODE_TYPE default from e2-custom-6-16384 to e2-standard-4 for all GCP jobs (serial and non-serial). The same openshift#82856 campaign found no disqualifying etcd/CPU regression at this size across 3 rounds. - e2e-gcp-custom-dns (nightly-4.21/4.22/5.0): replaces the only 3 pre-existing CONTROL_PLANE_NODE_TYPE/COMPUTE_NODE_TYPE overrides in the repo, e2-custom-8-32768, with e2-standard-8. Both are 8 vCPU/32GB, but GCP's custom machine type pricing carries a ~5% premium over the equivalent predefined shape when the ratio matches a predefined tier exactly (confirmed via the GCP Cloud Billing Catalog API: $616.37/mo vs $587.02/mo for 3 nodes). Strict cost win, no behavior change. A further step-down to n4-standard-4 was also evaluated (openshift#82929, openshift#82934) but is not currently viable: some installer versions reject n4-standard-4's required hyperdisk-balanced disk type outright, and instances (workers, and in one case a control-plane node) intermittently vanished mid-provisioning with "can't find created instance" - consistent with GCP-side capacity/stockout behavior on the n4 family in the tested zones rather than a config issue. Revisit in the future once GCP capacity/tooling matures. Scope: only the shared *-case (amd64) defaults in ipi-conf-gcp-commands.sh are touched; xlarge/large/compact SIZE_VARIANT branches and the arm64 path (t2a-standard-4, unvalidated by this work) are untouched. Pre-existing per-job overrides elsewhere in the repo (e.g. jobs pinned to larger sizes for their own reasons) are left alone. rh-pre-commit.version: 2.4.0 rh-pre-commit.check-secrets: ENABLED
|
/pj-rehearse periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-gcp-ovn-rt-upgrade periodic-ci-openshift-release-main-ci-5.0-e2e-gcp-ovn periodic-ci-openshift-release-main-nightly-5.0-e2e-gcp-ovn-rt periodic-ci-openshift-release-main-nightly-5.0-e2e-gcp-ovn-serial periodic-ci-openshift-release-main-ci-5.0-e2e-gcp-ovn-techpreview periodic-ci-openshift-release-main-ci-5.0-e2e-gcp-ovn-techpreview-serial-1of2 periodic-ci-openshift-release-main-ci-5.0-e2e-gcp-ovn-techpreview-serial-2of2 periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-gcp-ovn-upgrade periodic-ci-openshift-release-main-ci-5.0-e2e-gcp-ovn-upgrade periodic-ci-openshift-release-main-ci-4.22-upgrade-from-stable-4.21-e2e-gcp-ovn-rt-upgrade |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository YAML (base), Central YAML (inherited) Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (34)
WalkthroughThe change updates GCP machine type defaults and adds explicit control-plane machine type overrides to release and nightly serial test configurations across multiple OpenShift versions. ChangesGCP machine type configuration
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
Suggested labels: 🚥 Pre-merge checks | ✅ 15✅ Passed checks (15 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: sdodson The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
[REHEARSALNOTIFIER]
A total of 6218 jobs have been affected by this change. The above listing is non-exhaustive and limited to 25 jobs. A full list of affected jobs can be found here Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
|
@sdodson: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-openshift-release-main-ci-5.0-e2e-gcp-ovn-techpreview-serial-1of2 |
|
@sdodson: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-openshift-release-main-ci-5.0-e2e-gcp-ovn periodic-ci-openshift-release-main-ci-5.0-e2e-gcp-ovn-techpreview periodic-ci-openshift-release-main-ci-5.0-e2e-gcp-ovn-techpreview-serial-2of2 periodic-ci-openshift-release-main-ci-5.0-e2e-gcp-ovn-upgrade periodic-ci-openshift-release-main-nightly-5.0-e2e-gcp-ovn-rt |
|
@sdodson: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-openshift-release-main-ci-4.22-upgrade-from-stable-4.21-e2e-gcp-ovn-rt-upgrade periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-gcp-ovn-rt-upgrade periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-gcp-ovn-upgrade periodic-ci-openshift-release-main-nightly-5.0-e2e-gcp-ovn-serial |
|
@sdodson: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-openshift-release-main-ci-5.0-e2e-gcp-ovn-techpreview-serial-1of2 |
|
@sdodson: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-openshift-release-main-ci-4.22-upgrade-from-stable-4.21-e2e-gcp-ovn-rt-upgrade |
|
@sdodson: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-openshift-release-main-ci-5.0-e2e-gcp-ovn-techpreview periodic-ci-openshift-release-main-ci-5.0-e2e-gcp-ovn periodic-ci-openshift-release-main-nightly-5.0-e2e-gcp-ovn-rt |
|
@sdodson: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-openshift-release-main-ci-5.0-e2e-gcp-ovn-techpreview-serial-2of2 periodic-ci-openshift-release-main-ci-5.0-e2e-gcp-ovn-upgrade |
|
@sdodson: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-openshift-release-main-ci-5.0-e2e-gcp-ovn-techpreview-serial-1of2 periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-gcp-ovn-upgrade periodic-ci-openshift-release-main-nightly-5.0-e2e-gcp-ovn-serial |
|
@sdodson: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-gcp-ovn-rt-upgrade |
|
@sdodson: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-openshift-release-main-ci-5.0-e2e-gcp-ovn periodic-ci-openshift-release-main-nightly-5.0-e2e-gcp-ovn-rt periodic-ci-openshift-release-main-ci-5.0-e2e-gcp-ovn-techpreview |
|
@sdodson: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-openshift-release-main-ci-4.22-upgrade-from-stable-4.21-e2e-gcp-ovn-rt-upgrade |
|
@sdodson: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
@sdodson: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-openshift-release-main-nightly-5.0-e2e-gcp-ovn-serial |
|
@sdodson: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-openshift-release-main-nightly-5.0-e2e-gcp-ovn-serial |
|
@sdodson: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-openshift-release-main-ci-5.0-e2e-gcp-ovn-techpreview-serial-2of2 |
|
@sdodson: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-openshift-release-main-ci-4.22-upgrade-from-stable-4.21-e2e-gcp-ovn-rt-upgrade |
|
@sdodson: your |
|
/pj-rehearse periodic-ci-openshift-release-main-ci-4.22-upgrade-from-stable-4.21-e2e-gcp-ovn-rt-upgrade |
|
@sdodson: your |
|
/pj-rehearse periodic-ci-openshift-release-main-ci-4.22-upgrade-from-stable-4.21-e2e-gcp-ovn-rt-upgrade |
|
@sdodson: your |
|
/pj-rehearse periodic-ci-openshift-release-main-nightly-5.0-e2e-gcp-ovn-serial |
|
/pj-rehearse periodic-ci-openshift-release-main-ci-4.22-upgrade-from-stable-4.21-e2e-gcp-ovn-rt-upgrade |
|
@sdodson: your |
1 similar comment
|
@sdodson: your |
|
/pj-rehearse periodic-ci-openshift-release-main-ci-4.22-upgrade-from-stable-4.21-e2e-gcp-ovn-rt-upgrade |
|
@sdodson: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-openshift-release-main-nightly-5.0-e2e-gcp-ovn-serial |
|
/pj-rehearse periodic-ci-openshift-release-main-ci-5.0-e2e-gcp-ovn-techpreview-serial-2of2 |
|
@sdodson: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
1 similar comment
|
@sdodson: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-openshift-release-main-ci-4.22-upgrade-from-stable-4.21-e2e-gcp-ovn-rt-upgrade |
|
@sdodson: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
@sdodson: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
/close |
|
@sdodson: Closed this PR. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
Summary
Rehearsal-only validation PR (⚠️ DO NOT MERGE), identical diff to #82852.
Purpose: gather enough rehearsal runs on the 10 GCP jobs where a two-proportion power calculation (90% confidence, 80% power, historical 30-day Sippy pass rate as baseline) makes it statistically tractable (<30 new runs) to either prove or disprove a ≥50% reduction in failure rate under the new sizing scheme (e2-custom-8-16384 non-serial CP / e2-custom-6-16384 serial CP / e2-standard-4 workers).
Kept on a separate branch/PR from #82852 so the real production PR doesn't accumulate dozens of rehearsal-trigger comments.
Summary by CodeRabbit
This rehearsal updates OpenShift CI GCP jobs across release and nightly configurations.
e2-custom-6-16384control-plane nodes.e2-custom-8-16384control-plane nodes ande2-standard-4compute nodes.e2-standard-8for compute and control-plane nodes.