Repository navigation
gcp: change default control-plane instance size from custom-6-16384 to custom-8-16384 - #82852
Conversation
openshift-ci
Bot
added
the
needs-rebase
WalkthroughGCP provisioning defaults now use updated master and compute machine types. CI and nightly GCP serial test configurations explicitly set control-plane machine types. Custom DNS tests use ChangesGCP machine type configuration
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 15✅ Passed checks (15 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
@sdodson, Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@ci-operator/step-registry/ipi/conf/gcp/ipi-conf-gcp-commands.sh`:
- Around line 41-44: Add representative AMD64 GCP job links and explicit
sign-off evidence for the e2-custom-8-16384 default in the surrounding comment
for master_type_suffix. Retain the existing PR reference and performance
rationale, and ensure the cited runs cover the exact instance type before merge.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: d6eb6e1b-93fb-4b25-af3c-18060c46090a
📒 Files selected for processing (12)
ci-operator/config/openshift/openshift-tests-private/openshift-openshift-tests-private-release-4.22.yamlci-operator/config/openshift/openshift-tests-private/openshift-openshift-tests-private-release-4.22__amd64-nightly-4.22-upgrade-from-stable-4.21.yamlci-operator/config/openshift/openshift-tests-private/openshift-openshift-tests-private-release-4.22__amd64-nightly-4.22-upgrade-from-stable-4.22.yamlci-operator/config/openshift/openshift-tests-private/openshift-openshift-tests-private-release-4.22__amd64-nightly.yamlci-operator/config/openshift/openshift-tests-private/openshift-openshift-tests-private-release-4.22__amd64-rollback-nightly.yamlci-operator/config/openshift/openshift-tests-private/openshift-openshift-tests-private-release-4.22__amd64-stable-4.22-upgrade-from-stable-4.21.yamlci-operator/config/openshift/openshift-tests-private/openshift-openshift-tests-private-release-4.22__multi-nightly-4.22-cpou-upgrade-from-4.20.yamlci-operator/config/openshift/openshift-tests-private/openshift-openshift-tests-private-release-4.22__multi-nightly-4.22-upgrade-from-stable-4.21.yamlci-operator/config/openshift/openshift-tests-private/openshift-openshift-tests-private-release-4.22__multi-nightly-4.22-upgrade-from-stable-4.22.yamlci-operator/config/openshift/openshift-tests-private/openshift-openshift-tests-private-release-4.22__multi-nightly.yamlci-operator/config/openshift/openshift-tests-private/openshift-openshift-tests-private-release-4.22__multi-stable.yamlci-operator/step-registry/ipi/conf/gcp/ipi-conf-gcp-commands.sh
| # 8 vCPU / 16GB default: empirically validated (PR openshift/release#82754) to | ||
| # eliminate the CPU saturation / etcd disk-latency pattern seen with the prior | ||
| # 6 vCPU default, at a lower cost than e2-standard-8. | ||
| master_type_suffix="custom-8-16384" |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
rg -n -C 8 \
'master_type_suffix|SIZE_VARIANT|custom-6-16384|custom-8-16384' \
ci-operator/step-registry/ipi/conf/gcp \
ci-operator/configRepository: openshift/release
Length of output: 2110
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
printf '%s\n' '--- repository state ---'
pwd
git status --short --branch
printf '%s\n' '--- target files ---'
git ls-files 'ci-operator/step-registry/ipi/conf/gcp/ipi-conf-gcp-commands.sh' 'ci-operator/config/**' | head -80
printf '%s\n' '--- relevant references ---'
rg -n -C 8 'master_type_suffix|SIZE_VARIANT|custom-6-16384|custom-8-16384' \
ci-operator/step-registry/ipi/conf/gcp ci-operator/configRepository: openshift/release
Length of output: 6742
🌐 Web query:
openshift/release PR 82754 GCP custom-8-16384 empirical validation
💡 Result:
There is no evidence of a pull request numbered 82754 in the openshift/release repository [1][2][3][4]. The number 82754 is not associated with an OpenShift release or an empirical validation task for the GCP custom machine type custom-8-16384. While custom-8-16384 refers to a standard Google Cloud Platform (GCP) custom instance type configuration with 8 vCPUs and 16 GiB (16384 MB) of memory [5][6], it is a generic resource specification rather than a specific OpenShift validation target linked to the requested PR number [7]. Publicly available information regarding 82754 unrelated to OpenShift includes references to technical documentation for hardware components [8], security configuration guidelines [9][10], and unrelated software patches [11].
Citations:
- 1: c266a73
- 2: 5996017
- 3: 8514e4e
- 4: a30a9ba
- 5: https://github.com/openshift/openshift-docs/blob/3c1ab270/modules/sdpolicy-am-gcp-compute-types.adoc
- 6: https://discuss.ray.io/t/gcp-custom-machine-type-and-best-practices-for-node-choice/4732
- 7: openshift/installer@f8ab1fb...6723dfd
- 8: https://eu.raptorsupplies.com/pd/volkel/82754
- 9: https://ato-pathways.com/catalogs/references/cce/CCE-82754-3
- 10: https://static.open-scap.org/ssg-guides/ssg-rhcos4-guide-anssi_bp28_intermediary.html
- 11: http://mails.dpdk.org/archives/test-report/2020-October/163416.html
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
printf '%s\n' '--- target script ---'
sed -n '1,100p' ci-operator/step-registry/ipi/conf/gcp/ipi-conf-gcp-commands.sh
printf '%s\n' '--- repository references ---'
rg -n -C 8 'master_type_suffix|SIZE_VARIANT|custom-6-16384|custom-8-16384' \
ci-operator/step-registry/ipi/conf/gcp ci-operator/config || true
printf '%s\n' '--- related GCP step files ---'
git ls-files 'ci-operator/step-registry/ipi/conf/gcp/*'Repository: openshift/release
Length of output: 50374
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
for url in \
'https://api.github.com/repos/openshift/release/pulls/82754' \
'https://api.github.com/repos/openshift/release/issues/82754' \
'https://api.github.com/repos/openshift/release/commits?per_page=100'
do
printf '\n--- %s ---\n' "$url"
curl -LfsS --max-time 20 \
-H 'Accept: application/vnd.github+json' \
"$url" | jq '{message,number,title,state,html_url,merged_at,body}'
doneRepository: openshift/release
Length of output: 274
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
printf '%s\n' '--- target script ---'
cat -n ci-operator/step-registry/ipi/conf/gcp/ipi-conf-gcp-commands.sh
printf '%s\n' '--- exact symbol references in step registry ---'
rg -n -C 5 'master_type_suffix|custom-6-16384|custom-8-16384' ci-operator/step-registry || true
printf '%s\n' '--- GCP workflow and chain references ---'
rg -n -C 4 'ipi-conf-gcp|ipi-gcp|conf-gcp' ci-operator/step-registry ci-operator/config --glob '*.yaml' --glob '*.json' || true
printf '%s\n' '--- cited identifier in local repository ---'
rg -n '82754' . || trueRepository: openshift/release
Length of output: 50374
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
printf '%s\n' '--- recent history for target script ---'
git log -n 12 --format='%h %ad %s' --date=short -- ci-operator/step-registry/ipi/conf/gcp/ipi-conf-gcp-commands.sh
printf '%s\n' '--- repository remotes ---'
git remote -vRepository: openshift/release
Length of output: 441
🌐 Web query:
site:github.com/openshift/release/pull/82754
💡 Result:
PR #82754 in openshift/release is an open, rehearsal-only experiment titled:
[Debug] Test e2-standard-8 control plane for gcp-ovn-rt-upgrade
It tests larger GCP control-plane nodes to determine whether additional CPU and disk throughput reduce etcd latency and CPU-saturation issues. It should not be merged. The PR later also tests n2-standard-4 as a lower-sizing comparison. [1] (github.com)
Citations:
🌐 Web query:
"openshift/release#82754"
💡 Result:
There is no public information available regarding a pull request with the number #82754 in the openshift/release repository. Search results for this number returned references to a Common Configuration Enumeration (CCE) identifier, CCE-82754-3, associated with the secure configuration of Red Hat Enterprise Linux CoreOS rather than an OpenShift pull request [1][2]. Other pull requests in the openshift/release repository exist with different identifiers [3][4][5][6][7].
Citations:
- 1: https://static.open-scap.org/ssg-guides/ssg-rhcos4-guide-anssi_bp28_intermediary.html
- 2: https://complianceascode.github.io/content-pages/guides/ssg-rhcos4-guide-anssi_bp28_intermediary.html
- 3: 25e5022
- 4: 8844aed
- 5: 5996017
- 6: 23ab27d
- 7: CNTRLPLANE-3509: Deep-verify release component digests in nodepool resolve step #80576
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
printf '%s\n' '--- current fallback logic ---'
sed -n '1,90p' ci-operator/step-registry/ipi/conf/gcp/ipi-conf-gcp-commands.sh
printf '%s\n' '--- change commit ---'
git show --format=fuller --no-ext-diff --unified=25 \
2ca8c5d90 -- ci-operator/step-registry/ipi/conf/gcp/ipi-conf-gcp-commands.sh
printf '%s\n' '--- exact machine-size references in GCP step files ---'
rg -n 'master_type_suffix|custom-[0-9]+-[0-9]+|e2-standard-8|n2-standard-4' \
ci-operator/step-registry/ipi/conf/gcpRepository: openshift/release
Length of output: 12924
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
printf '%s\n' '--- commit and parent metadata ---'
git show -s --format='%H%n%P%n%B' 2ca8c5d90
printf '%s\n' '--- parent value ---'
git show 2ca8c5d90^:ci-operator/step-registry/ipi/conf/gcp/ipi-conf-gcp-commands.sh \
| sed -n '30,55p'Repository: openshift/release
Length of output: 1698
🌐 Web query:
site:github.com/openshift/release/pull/82754 "custom-8-16384"
💡 Result:
No search results were found for:
site:github.com/openshift/release/pull/82754 "custom-8-16384"
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
printf '%s\n' '--- control-plane override references ---'
rg -n -C 5 'CONTROL_PLANE_NODE_TYPE|e2-custom-8-32768' \
ci-operator/step-registry/ipi ci-operator/config || true
printf '%s\n' '--- files with the claimed override ---'
rg -l 'e2-custom-8-32768' ci-operator/config ci-operator/step-registry || true
printf '%s\n' '--- default-size inputs in the GCP configuration path ---'
rg -n -C 4 'SIZE_VARIANT|CONTROL_ARCH|OCP_ARCH|COMPUTE_NODE_TYPE' \
ci-operator/step-registry/ipi/conf/gcp \
ci-operator/step-registry/ipi/conf \
--glob '*.yaml' --glob '*.sh' || trueRepository: openshift/release
Length of output: 50374
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
needles = (
"chain: ipi-conf-gcp",
"ref: ipi-conf-gcp",
"workflow: ipi-gcp",
"e2-custom-8-32768",
"CONTROL_PLANE_NODE_TYPE",
)
for root in (Path("ci-operator/config"), Path("ci-operator/step-registry")):
counts = {needle: 0 for needle in needles}
files = {needle: set() for needle in needles}
for path in root.rglob("*"):
if not path.is_file():
continue
try:
text = path.read_text(errors="replace")
except OSError:
continue
for needle in needles:
if needle in text:
counts[needle] += text.count(needle)
files[needle].add(str(path))
print(root)
for needle in needles:
print(f"{needle}: {counts[needle]} matches in {len(files[needle])} files")
for path in sorted(files[needle])[:20]:
print(f" {path}")
PYRepository: openshift/release
Length of output: 18284
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
printf '%s\n' '--- explicit control-plane override step ---'
cat -n ci-operator/step-registry/ipi/conf/gcp/control-plane-node-type/ipi-conf-gcp-control-plane-node-type-commands.sh
cat -n ci-operator/step-registry/ipi/conf/gcp/control-plane-node-type/ipi-conf-gcp-control-plane-node-type-ref.yaml
printf '%s\n' '--- three claimed override jobs ---'
rg -n -C 10 'e2-custom-8-32768|ipi-conf-gcp-control-plane-node-type' \
ci-operator/config/openshift/release/openshift-release-main__nightly-4.21.yaml \
ci-operator/config/openshift/release/openshift-release-main__nightly-4.22.yaml \
ci-operator/config/openshift/release/openshift-release-main__nightly-5.0.yamlRepository: openshift/release
Length of output: 10103
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
printf '%s\n' '--- default GCP provision chain ---'
cat -n ci-operator/step-registry/cucushift/installer/rehearse/gcp/ipi/default/provision/cucushift-installer-rehearse-gcp-ipi-default-provision-chain.yaml
printf '%s\n' '--- GCP machine-type check ---'
cat -n ci-operator/step-registry/cucushift/installer/check/gcp/machine-type/cucushift-installer-check-gcp-machine-type-commands.shRepository: openshift/release
Length of output: 3733
Link the representative validation and sign-off for e2-custom-8-16384.
PR openshift/release#82754 is cited for 12 rehearsal runs, including this exact type, but no run links or sign-off are included. Add evidence for representative AMD64 GCP jobs before merge.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@ci-operator/step-registry/ipi/conf/gcp/ipi-conf-gcp-commands.sh` around lines
41 - 44, Add representative AMD64 GCP job links and explicit sign-off evidence
for the e2-custom-8-16384 default in the surrounding comment for
master_type_suffix. Retain the existing PR reference and performance rationale,
and ensure the cited runs cover the exact instance type before merge.
|
@sdodson, Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
8ec7704 to
5103c9d
Compare
openshift-ci
Bot
removed
the
needs-rebase
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
|
[REHEARSALNOTIFIER]
A total of 6218 jobs have been affected by this change. The above listing is non-exhaustive and limited to 25 jobs. A full list of affected jobs can be found here Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
…planes, e2-standard-4 workers Final production GCP sizing scheme, based on two rehearsal campaigns (openshift#82754, openshift#82856): - Control planes: change the shared ipi-conf-gcp default from e2-custom-6-16384 (6 vCPU / 16GB) to e2-custom-8-16384 (8 vCPU / 16GB). The prior default carried a code comment noting it was a "temporary test to see if this helps the consistent high CPU alerts and random test failures" - confirmed via a 12-run rehearsal campaign across 3 alternative instance types (openshift#82754) and a 30-day production survey across ~20 GCP 5.0 jobs, both showing e2-custom-8-16384 matches or beats e2-standard-8 on etcd fsync/commit latency and CPU saturation at ~13% lower cost. vCPU count (not memory ratio or CPU host platform) was the dominant driver: n2-standard-4 (4 vCPU) performed worse than the 6 vCPU baseline, every 8-vCPU option performed better. - Serial control planes: pinned back down to e2-custom-6-16384 via an explicit CONTROL_PLANE_NODE_TYPE override on every serial GCP job (51 jobs across 33 config files, using workflows openshift-e2e-gcp-serial{,-runc,-cgroupsv1}, openshift-e2e-gcp-ovn-serial, openshift-e2e-gcp-sdn-serial, openshift-e2e-gcp-techpreview-serial). Serial jobs run a non-parallel test suite and already show lower control-plane CPU utilization than non-serial jobs, and a 3-round study (openshift#82856) found downsizing them further to e2-standard-4 (4 vCPU) causes a clear, reproducible etcd disk-latency regression (fsync p99 elevated 29-113% above baseline in every round with data, including one round with real etcd overload errors: "etcdserver: too many requests", leader elections). A handful of old (nightly-4.6-4.16) jobs have "serial" in their as: name but run the parallel workflow (openshift-e2e-gcp/openshift-e2e-gcp-sdn) - left alone, since serial-ness is determined by workflow, not job name. - Workers: change the shared COMPUTE_NODE_TYPE default from e2-custom-6-16384 to e2-standard-4 for all GCP jobs (serial and non-serial). The same openshift#82856 campaign found no disqualifying etcd/CPU regression at this size across 3 rounds. - e2e-gcp-custom-dns (nightly-4.21/4.22/5.0): replaces the only 3 pre-existing CONTROL_PLANE_NODE_TYPE/COMPUTE_NODE_TYPE overrides in the repo, e2-custom-8-32768, with e2-standard-8. Both are 8 vCPU/32GB, but GCP's custom machine type pricing carries a ~5% premium over the equivalent predefined shape when the ratio matches a predefined tier exactly (confirmed via the GCP Cloud Billing Catalog API: $616.37/mo vs $587.02/mo for 3 nodes). Strict cost win, no behavior change. A further step-down to n4-standard-4 was also evaluated (openshift#82929, openshift#82934) but is not currently viable: some installer versions reject n4-standard-4's required hyperdisk-balanced disk type outright, and instances (workers, and in one case a control-plane node) intermittently vanished mid-provisioning with "can't find created instance" - consistent with GCP-side capacity/stockout behavior on the n4 family in the tested zones rather than a config issue. Revisit in the future once GCP capacity/tooling matures. Scope: only the shared *-case (amd64) defaults in ipi-conf-gcp-commands.sh are touched; xlarge/large/compact SIZE_VARIANT branches and the arm64 path (t2a-standard-4, unvalidated by this work) are untouched. Pre-existing per-job overrides elsewhere in the repo (e.g. jobs pinned to larger sizes for their own reasons) are left alone. rh-pre-commit.version: 2.4.0 rh-pre-commit.check-secrets: ENABLED
b789b7e to
436ae3f
Compare
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
|
@sdodson: |
|
@sdodson: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: dgoodwin, sdodson The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
Bot
added
the
approved
|
/pj-rehearse ack |
|
@sdodson: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
openshift-merge-bot
Bot
added
the
rehearsals-ack
…r serial GCP jobs DO NOT MERGE - rehearsal-only debug PR. Follow-up to openshift#82852 (production default bump to e2-custom-8-16384) and openshift#82856 (e2-standard-4 study, now concluded: workers showed no consistent latency regression but a plateaued increase in worker CPU>95% breach time on some jobs; serial control planes showed a clear, reproducible etcd disk-latency regression and are not recommended at e2-standard-4). This PR studies n4-standard-4 (Intel Emerald Rapids) as an alternative sizing point, using the same 8-job structure: 4 non-serial jobs get COMPUTE_NODE_TYPE=n4-standard-4 (workers only); 4 serial jobs get both COMPUTE_NODE_TYPE and CONTROL_PLANE_NODE_TYPE=n4-standard-4. rh-pre-commit.version: 2.4.0 rh-pre-commit.check-secrets: ENABLED
…planes, e2-standard-4 workers (openshift#82852) Final production GCP sizing scheme, based on two rehearsal campaigns (openshift#82754, openshift#82856): - Control planes: change the shared ipi-conf-gcp default from e2-custom-6-16384 (6 vCPU / 16GB) to e2-custom-8-16384 (8 vCPU / 16GB). The prior default carried a code comment noting it was a "temporary test to see if this helps the consistent high CPU alerts and random test failures" - confirmed via a 12-run rehearsal campaign across 3 alternative instance types (openshift#82754) and a 30-day production survey across ~20 GCP 5.0 jobs, both showing e2-custom-8-16384 matches or beats e2-standard-8 on etcd fsync/commit latency and CPU saturation at ~13% lower cost. vCPU count (not memory ratio or CPU host platform) was the dominant driver: n2-standard-4 (4 vCPU) performed worse than the 6 vCPU baseline, every 8-vCPU option performed better. - Serial control planes: pinned back down to e2-custom-6-16384 via an explicit CONTROL_PLANE_NODE_TYPE override on every serial GCP job (51 jobs across 33 config files, using workflows openshift-e2e-gcp-serial{,-runc,-cgroupsv1}, openshift-e2e-gcp-ovn-serial, openshift-e2e-gcp-sdn-serial, openshift-e2e-gcp-techpreview-serial). Serial jobs run a non-parallel test suite and already show lower control-plane CPU utilization than non-serial jobs, and a 3-round study (openshift#82856) found downsizing them further to e2-standard-4 (4 vCPU) causes a clear, reproducible etcd disk-latency regression (fsync p99 elevated 29-113% above baseline in every round with data, including one round with real etcd overload errors: "etcdserver: too many requests", leader elections). A handful of old (nightly-4.6-4.16) jobs have "serial" in their as: name but run the parallel workflow (openshift-e2e-gcp/openshift-e2e-gcp-sdn) - left alone, since serial-ness is determined by workflow, not job name. - Workers: change the shared COMPUTE_NODE_TYPE default from e2-custom-6-16384 to e2-standard-4 for all GCP jobs (serial and non-serial). The same openshift#82856 campaign found no disqualifying etcd/CPU regression at this size across 3 rounds. - e2e-gcp-custom-dns (nightly-4.21/4.22/5.0): replaces the only 3 pre-existing CONTROL_PLANE_NODE_TYPE/COMPUTE_NODE_TYPE overrides in the repo, e2-custom-8-32768, with e2-standard-8. Both are 8 vCPU/32GB, but GCP's custom machine type pricing carries a ~5% premium over the equivalent predefined shape when the ratio matches a predefined tier exactly (confirmed via the GCP Cloud Billing Catalog API: $616.37/mo vs $587.02/mo for 3 nodes). Strict cost win, no behavior change. A further step-down to n4-standard-4 was also evaluated (openshift#82929, openshift#82934) but is not currently viable: some installer versions reject n4-standard-4's required hyperdisk-balanced disk type outright, and instances (workers, and in one case a control-plane node) intermittently vanished mid-provisioning with "can't find created instance" - consistent with GCP-side capacity/stockout behavior on the n4 family in the tested zones rather than a config issue. Revisit in the future once GCP capacity/tooling matures. Scope: only the shared *-case (amd64) defaults in ipi-conf-gcp-commands.sh are touched; xlarge/large/compact SIZE_VARIANT branches and the arm64 path (t2a-standard-4, unvalidated by this work) are untouched. Pre-existing per-job overrides elsewhere in the repo (e.g. jobs pinned to larger sizes for their own reasons) are left alone. rh-pre-commit.version: 2.4.0 rh-pre-commit.check-secrets: ENABLED
2 participants
Summary
Final production GCP sizing scheme for CI, based on three rehearsal campaigns (#82754, #82856, #83005).
e2-custom-6-16384→e2-custom-8-16384(+2 vCPU)CONTROL_PLANE_NODE_TYPE: e2-custom-6-16384override added (51 jobs / 33 config files) to opt out of the new shared defaulte2-custom-6-16384→e2-standard-4e2-custom-6-16384→e2-standard-4(same change as non-serial)Also:
e2e-gcp-custom-dns(nightly-4.21/4.22/5.0) movese2-custom-8-32768→e2-standard-8— identical specs, avoids GCP's custom-machine-type pricing premium. Cost-only, no behavior change.Why
e2-standard-4did regress etcd fsync p99 (29-113% above baseline, with one round hitting etcd overload errors), so serial control planes stay at their current 6-vCPU size — neither upsized nor downsized.e2-standard-4(all jobs): validated safe, not beneficial — no latency or correctness regression on any job class across all rehearsal rounds in the earlier campaign. This is a cost reduction with no measured downside, not a reliability fix.n4-standard-4was evaluated and shelved: some installer versions reject its required disk type, and instances intermittently vanished mid-provisioning (GCP capacity/stockout behavior on the n4 family, not a config issue).Validation results (#83005)
10 production GCP jobs (2 release-blocking, 8 informing) rehearsed against fixed 30-day Sippy baselines, using a two-proportion power test (90% confidence / 80% power) for a ≥50% failure-rate reduction. All 10 jobs PROVEN, including both blocking jobs:
Scope
*-case (amd64) defaults inipi-conf-gcp-commands.shchange.SIZE_VARIANTbranches (xlarge/large/compact),arm64, and pre-existing per-job overrides are untouched.CONTROL_PLANE_NODE_TYPEoverride applies only to jobs on a genuinely serial workflow (openshift-e2e-gcp-serial*,openshift-e2e-gcp-ovn-serial,openshift-e2e-gcp-sdn-serial,openshift-e2e-gcp-techpreview-serial). A few old jobs with "serial" in theas:name but running the parallel workflow are intentionally left alone.Test plan
e2e-gcp-custom-dnsto confirme2-standard-8behaves identically toe2-custom-8-32768e2family, only changes size)