Repository navigation
Conversation
|
/pj-rehearse periodic-ci-openshift-release-main-ci-5.0-e2e-gcp-ovn-rhcos10-techpreview periodic-ci-openshift-release-main-ci-4.15-e2e-gcp-ovn-techpreview periodic-ci-openshift-release-main-nightly-4.22-e2e-gcp-ovn-runc periodic-ci-openshift-release-main-nightly-5.0-e2e-gcp-ovn-runc periodic-ci-openshift-release-main-ci-4.15-e2e-gcp-ovn-techpreview-serial periodic-ci-openshift-release-main-nightly-4.22-e2e-gcp-ovn-serial-runc periodic-ci-openshift-release-main-nightly-5.0-e2e-gcp-ovn-serial-runc periodic-ci-openshift-release-main-nightly-5.0-e2e-gcp-ovn-serial |
|
@sdodson: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository YAML (base), Central YAML (inherited) Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (4)
🚧 Files skipped from review as they are similar to previous changes (4)
Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review. WalkthroughThe pull request updates selected GCP test configurations. Compute and control-plane nodes use ChangesGCP node configuration
Estimated code review effort: 1 (Trivial) | ~5 minutes Merge Risk: ⚪ Minimal · up to This PR changes configuration for a rehearsal-only n4-standard-4 sizing study; no actionable merge-blocking risk remains in the supplied evidence. Suggested reviewers: 🚥 Pre-merge checks | ✅ 15✅ Passed checks (15 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: sdodson The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
openshift-ci
Bot
added
the
approved
|
/pj-rehearse periodic-ci-openshift-release-main-ci-4.15-e2e-gcp-ovn-techpreview-serial periodic-ci-openshift-release-main-nightly-4.22-e2e-gcp-ovn-serial-runc periodic-ci-openshift-release-main-nightly-5.0-e2e-gcp-ovn-serial-runc periodic-ci-openshift-release-main-nightly-5.0-e2e-gcp-ovn-serial |
|
@sdodson: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-openshift-release-main-ci-5.0-e2e-gcp-ovn-rhcos10-techpreview periodic-ci-openshift-release-main-ci-4.15-e2e-gcp-ovn-techpreview periodic-ci-openshift-release-main-nightly-4.22-e2e-gcp-ovn-runc periodic-ci-openshift-release-main-nightly-5.0-e2e-gcp-ovn-runc periodic-ci-openshift-release-main-nightly-4.22-e2e-gcp-ovn-serial-runc periodic-ci-openshift-release-main-nightly-5.0-e2e-gcp-ovn-serial-runc periodic-ci-openshift-release-main-nightly-5.0-e2e-gcp-ovn-serial |
|
@sdodson: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-openshift-release-main-ci-4.15-e2e-gcp-ovn-techpreview periodic-ci-openshift-release-main-nightly-5.0-e2e-gcp-ovn-serial |
|
@sdodson: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
Closing — n4-standard-4 is not currently viable for GCP CI: (1) control-plane disk type defaults to pd-ssd which n4-standard-4 rejects (requires hyperdisk-balanced); (2) older installer versions (e.g. ci-4.15's) don't even recognize hyperdisk-balanced as a valid value; (3) even where fixed, every completed job in the final round showed at least one instance (worker, and in one case the control plane itself) vanish mid-provisioning with "can't find created instance" — consistent with GCP-side capacity/stockout behavior on the n4 family in the tested zones, not a config bug. Shelving n4-standard-4 for a future revisit. Production sizing scheme (e2-custom-8-16384 non-serial CP / e2-custom-6-16384 serial CP / e2-standard-4 workers) materialized in #82852. |
…e2-custom-6-16384 Final production sizing scheme, based on the concluded rehearsal campaigns in openshift#82754 (control-plane sizing) and openshift#82856 (e2-standard-4 worker/control-plane study): - Workers: change the shared ipi-conf-gcp default from e2-custom-6-16384 to e2-standard-4 for all GCP jobs (serial and non-serial). The study found no disqualifying etcd/CPU regression at this size. - Control planes: the shared default (already bumped to e2-custom-8-16384 in a prior commit) applies to non-serial jobs, which benefit from the extra headroom. SERIAL jobs run a non-parallel test suite and already show lower CPU utilization than non-serial jobs, and were found to regress when downsized further to e2-standard-4 (4 vCPU) - so every serial GCP job gets an explicit CONTROL_PLANE_NODE_TYPE override pinning it at the original e2-custom-6-16384, opting out of the new 8 vCPU default rather than paying for CPU headroom they don't need. n4-standard-4 was also evaluated as a further step-down (openshift#82929, openshift#82934) but is not currently viable: disk-type incompatibilities on some installer versions, and instances (both workers and, once, a control-plane node) intermittently vanishing mid-provisioning with "can't find created instance" - consistent with GCP-side capacity/stockout behavior on the n4 family. Revisit in the future once tooling/capacity matures. rh-pre-commit.version: 2.4.0 rh-pre-commit.check-secrets: ENABLED
…planes, e2-standard-4 workers Final production GCP sizing scheme, based on two rehearsal campaigns (openshift#82754, openshift#82856): - Control planes: change the shared ipi-conf-gcp default from e2-custom-6-16384 (6 vCPU / 16GB) to e2-custom-8-16384 (8 vCPU / 16GB). The prior default carried a code comment noting it was a "temporary test to see if this helps the consistent high CPU alerts and random test failures" - confirmed via a 12-run rehearsal campaign across 3 alternative instance types (openshift#82754) and a 30-day production survey across ~20 GCP 5.0 jobs, both showing e2-custom-8-16384 matches or beats e2-standard-8 on etcd fsync/commit latency and CPU saturation at ~13% lower cost. vCPU count (not memory ratio or CPU host platform) was the dominant driver: n2-standard-4 (4 vCPU) performed worse than the 6 vCPU baseline, every 8-vCPU option performed better. - Serial control planes: pinned back down to e2-custom-6-16384 via an explicit CONTROL_PLANE_NODE_TYPE override on every serial GCP job (51 jobs across 33 config files, using workflows openshift-e2e-gcp-serial{,-runc,-cgroupsv1}, openshift-e2e-gcp-ovn-serial, openshift-e2e-gcp-sdn-serial, openshift-e2e-gcp-techpreview-serial). Serial jobs run a non-parallel test suite and already show lower control-plane CPU utilization than non-serial jobs, and a 3-round study (openshift#82856) found downsizing them further to e2-standard-4 (4 vCPU) causes a clear, reproducible etcd disk-latency regression (fsync p99 elevated 29-113% above baseline in every round with data, including one round with real etcd overload errors: "etcdserver: too many requests", leader elections). A handful of old (nightly-4.6-4.16) jobs have "serial" in their as: name but run the parallel workflow (openshift-e2e-gcp/openshift-e2e-gcp-sdn) - left alone, since serial-ness is determined by workflow, not job name. - Workers: change the shared COMPUTE_NODE_TYPE default from e2-custom-6-16384 to e2-standard-4 for all GCP jobs (serial and non-serial). The same openshift#82856 campaign found no disqualifying etcd/CPU regression at this size across 3 rounds. - e2e-gcp-custom-dns (nightly-4.21/4.22/5.0): replaces the only 3 pre-existing CONTROL_PLANE_NODE_TYPE/COMPUTE_NODE_TYPE overrides in the repo, e2-custom-8-32768, with e2-standard-8. Both are 8 vCPU/32GB, but GCP's custom machine type pricing carries a ~5% premium over the equivalent predefined shape when the ratio matches a predefined tier exactly (confirmed via the GCP Cloud Billing Catalog API: $616.37/mo vs $587.02/mo for 3 nodes). Strict cost win, no behavior change. A further step-down to n4-standard-4 was also evaluated (openshift#82929, openshift#82934) but is not currently viable: some installer versions reject n4-standard-4's required hyperdisk-balanced disk type outright, and instances (workers, and in one case a control-plane node) intermittently vanished mid-provisioning with "can't find created instance" - consistent with GCP-side capacity/stockout behavior on the n4 family in the tested zones rather than a config issue. Revisit in the future once GCP capacity/tooling matures. Scope: only the shared *-case (amd64) defaults in ipi-conf-gcp-commands.sh are touched; xlarge/large/compact SIZE_VARIANT branches and the arm64 path (t2a-standard-4, unvalidated by this work) are untouched. Pre-existing per-job overrides elsewhere in the repo (e.g. jobs pinned to larger sizes for their own reasons) are left alone. rh-pre-commit.version: 2.4.0 rh-pre-commit.check-secrets: ENABLED
…planes, e2-standard-4 workers (#82852) Final production GCP sizing scheme, based on two rehearsal campaigns (#82754, #82856): - Control planes: change the shared ipi-conf-gcp default from e2-custom-6-16384 (6 vCPU / 16GB) to e2-custom-8-16384 (8 vCPU / 16GB). The prior default carried a code comment noting it was a "temporary test to see if this helps the consistent high CPU alerts and random test failures" - confirmed via a 12-run rehearsal campaign across 3 alternative instance types (#82754) and a 30-day production survey across ~20 GCP 5.0 jobs, both showing e2-custom-8-16384 matches or beats e2-standard-8 on etcd fsync/commit latency and CPU saturation at ~13% lower cost. vCPU count (not memory ratio or CPU host platform) was the dominant driver: n2-standard-4 (4 vCPU) performed worse than the 6 vCPU baseline, every 8-vCPU option performed better. - Serial control planes: pinned back down to e2-custom-6-16384 via an explicit CONTROL_PLANE_NODE_TYPE override on every serial GCP job (51 jobs across 33 config files, using workflows openshift-e2e-gcp-serial{,-runc,-cgroupsv1}, openshift-e2e-gcp-ovn-serial, openshift-e2e-gcp-sdn-serial, openshift-e2e-gcp-techpreview-serial). Serial jobs run a non-parallel test suite and already show lower control-plane CPU utilization than non-serial jobs, and a 3-round study (#82856) found downsizing them further to e2-standard-4 (4 vCPU) causes a clear, reproducible etcd disk-latency regression (fsync p99 elevated 29-113% above baseline in every round with data, including one round with real etcd overload errors: "etcdserver: too many requests", leader elections). A handful of old (nightly-4.6-4.16) jobs have "serial" in their as: name but run the parallel workflow (openshift-e2e-gcp/openshift-e2e-gcp-sdn) - left alone, since serial-ness is determined by workflow, not job name. - Workers: change the shared COMPUTE_NODE_TYPE default from e2-custom-6-16384 to e2-standard-4 for all GCP jobs (serial and non-serial). The same #82856 campaign found no disqualifying etcd/CPU regression at this size across 3 rounds. - e2e-gcp-custom-dns (nightly-4.21/4.22/5.0): replaces the only 3 pre-existing CONTROL_PLANE_NODE_TYPE/COMPUTE_NODE_TYPE overrides in the repo, e2-custom-8-32768, with e2-standard-8. Both are 8 vCPU/32GB, but GCP's custom machine type pricing carries a ~5% premium over the equivalent predefined shape when the ratio matches a predefined tier exactly (confirmed via the GCP Cloud Billing Catalog API: $616.37/mo vs $587.02/mo for 3 nodes). Strict cost win, no behavior change. A further step-down to n4-standard-4 was also evaluated (#82929, #82934) but is not currently viable: some installer versions reject n4-standard-4's required hyperdisk-balanced disk type outright, and instances (workers, and in one case a control-plane node) intermittently vanished mid-provisioning with "can't find created instance" - consistent with GCP-side capacity/stockout behavior on the n4 family in the tested zones rather than a config issue. Revisit in the future once GCP capacity/tooling matures. Scope: only the shared *-case (amd64) defaults in ipi-conf-gcp-commands.sh are touched; xlarge/large/compact SIZE_VARIANT branches and the arm64 path (t2a-standard-4, unvalidated by this work) are untouched. Pre-existing per-job overrides elsewhere in the repo (e.g. jobs pinned to larger sizes for their own reasons) are left alone. rh-pre-commit.version: 2.4.0 rh-pre-commit.check-secrets: ENABLED
|
/reopen |
|
@sdodson: Reopened this PR. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
openshift-ci
Bot
added
the
needs-rebase
…r serial GCP jobs DO NOT MERGE - rehearsal-only debug PR. Follow-up to openshift#82852 (production default bump to e2-custom-8-16384) and openshift#82856 (e2-standard-4 study, now concluded: workers showed no consistent latency regression but a plateaued increase in worker CPU>95% breach time on some jobs; serial control planes showed a clear, reproducible etcd disk-latency regression and are not recommended at e2-standard-4). This PR studies n4-standard-4 (Intel Emerald Rapids) as an alternative sizing point, using the same 8-job structure: 4 non-serial jobs get COMPUTE_NODE_TYPE=n4-standard-4 (workers only); 4 serial jobs get both COMPUTE_NODE_TYPE and CONTROL_PLANE_NODE_TYPE=n4-standard-4. rh-pre-commit.version: 2.4.0 rh-pre-commit.check-secrets: ENABLED
n4-standard-4 requires hyperdisk-balanced disks; the default CONTROL_PLANE_DISK_TYPE (pd-ssd) is incompatible and caused install to fail for all 4 serial-job control planes in round 1: controlPlane.platform.gcp.diskType: Invalid value: "pd-ssd": n4-standard-4 instance requires one of the following disk types: [hyperdisk-balanced] DO NOT MERGE - rehearsal-only debug PR. rh-pre-commit.version: 2.4.0 rh-pre-commit.check-secrets: ENABLED
3a4ba30 to
49298a2
Compare
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
openshift-ci
Bot
removed
the
needs-rebase
|
[REHEARSALNOTIFIER]
Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
|
/pj-rehearse periodic-ci-openshift-release-main-nightly-5.0-e2e-gcp-ovn-serial |
|
@sdodson: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
@sdodson: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
…planes, e2-standard-4 workers (openshift#82852) Final production GCP sizing scheme, based on two rehearsal campaigns (openshift#82754, openshift#82856): - Control planes: change the shared ipi-conf-gcp default from e2-custom-6-16384 (6 vCPU / 16GB) to e2-custom-8-16384 (8 vCPU / 16GB). The prior default carried a code comment noting it was a "temporary test to see if this helps the consistent high CPU alerts and random test failures" - confirmed via a 12-run rehearsal campaign across 3 alternative instance types (openshift#82754) and a 30-day production survey across ~20 GCP 5.0 jobs, both showing e2-custom-8-16384 matches or beats e2-standard-8 on etcd fsync/commit latency and CPU saturation at ~13% lower cost. vCPU count (not memory ratio or CPU host platform) was the dominant driver: n2-standard-4 (4 vCPU) performed worse than the 6 vCPU baseline, every 8-vCPU option performed better. - Serial control planes: pinned back down to e2-custom-6-16384 via an explicit CONTROL_PLANE_NODE_TYPE override on every serial GCP job (51 jobs across 33 config files, using workflows openshift-e2e-gcp-serial{,-runc,-cgroupsv1}, openshift-e2e-gcp-ovn-serial, openshift-e2e-gcp-sdn-serial, openshift-e2e-gcp-techpreview-serial). Serial jobs run a non-parallel test suite and already show lower control-plane CPU utilization than non-serial jobs, and a 3-round study (openshift#82856) found downsizing them further to e2-standard-4 (4 vCPU) causes a clear, reproducible etcd disk-latency regression (fsync p99 elevated 29-113% above baseline in every round with data, including one round with real etcd overload errors: "etcdserver: too many requests", leader elections). A handful of old (nightly-4.6-4.16) jobs have "serial" in their as: name but run the parallel workflow (openshift-e2e-gcp/openshift-e2e-gcp-sdn) - left alone, since serial-ness is determined by workflow, not job name. - Workers: change the shared COMPUTE_NODE_TYPE default from e2-custom-6-16384 to e2-standard-4 for all GCP jobs (serial and non-serial). The same openshift#82856 campaign found no disqualifying etcd/CPU regression at this size across 3 rounds. - e2e-gcp-custom-dns (nightly-4.21/4.22/5.0): replaces the only 3 pre-existing CONTROL_PLANE_NODE_TYPE/COMPUTE_NODE_TYPE overrides in the repo, e2-custom-8-32768, with e2-standard-8. Both are 8 vCPU/32GB, but GCP's custom machine type pricing carries a ~5% premium over the equivalent predefined shape when the ratio matches a predefined tier exactly (confirmed via the GCP Cloud Billing Catalog API: $616.37/mo vs $587.02/mo for 3 nodes). Strict cost win, no behavior change. A further step-down to n4-standard-4 was also evaluated (openshift#82929, openshift#82934) but is not currently viable: some installer versions reject n4-standard-4's required hyperdisk-balanced disk type outright, and instances (workers, and in one case a control-plane node) intermittently vanished mid-provisioning with "can't find created instance" - consistent with GCP-side capacity/stockout behavior on the n4 family in the tested zones rather than a config issue. Revisit in the future once GCP capacity/tooling matures. Scope: only the shared *-case (amd64) defaults in ipi-conf-gcp-commands.sh are touched; xlarge/large/compact SIZE_VARIANT branches and the arm64 path (t2a-standard-4, unvalidated by this work) are untouched. Pre-existing per-job overrides elsewhere in the repo (e.g. jobs pinned to larger sizes for their own reasons) are left alone. rh-pre-commit.version: 2.4.0 rh-pre-commit.check-secrets: ENABLED
|
/close |
|
@sdodson: Closed this PR. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
1 participant
Summary
Rehearsal-only study PR (⚠️ DO NOT MERGE), follow-up to:
e2-custom-8-16384Background: e2-standard-4 study findings (#82856)
Across 3 rehearsal rounds on the same 8 GCP jobs:
etcdserver: too many requests, leader elections, ReadIndex timeouts). Not recommended.This study
Testing
n4-standard-4(Intel Emerald Rapids) as an alternative sizing point using the identical 8-job structure and methodology:COMPUTE_NODE_TYPE: n4-standard-4(workers only)COMPUTE_NODE_TYPEandCONTROL_PLANE_NODE_TYPE: n4-standard-4Will trigger rehearsals to collect etcd fsync/commit latency, CPU>95% breach time, and pass/fail data across multiple rounds, comparing against both the original e2-custom-6-16384 baseline and the e2-standard-4 study's results.
Summary by CodeRabbit
This rehearsal-only PR updates eight GCP jobs across multiple OpenShift releases to use
n4-standard-4compute nodes. Serial jobs also usen4-standard-4control-plane nodes andhyperdisk-balancedcontrol-plane disks where required.