Windows Containers productization design by aravindhp · Pull Request #5 · openshift/enhancements

aravindhp · 2019-09-03T17:44:13Z

Initial enhancement proposal for productization of Windows Containers that will enable running of Windows workloads on an OpenShift cluster.

aravindhp · 2019-09-03T17:45:21Z

/cc @ravisantoshgudimetla @sdodson @crawford

sdodson

lgtm for 4.3, provides flexibility to adapt in the future

wking · 2019-09-10T16:10:06Z

+## Summary
+
+The intent of this enhancement is to allow a cluster administrator to add a
+Windows worker node with a prescribed configuration to an OpenShift cluster as a


nit: s/worker/compute/, e.g. see the old openshift/installer#1330. Overhauling existing wording it hard, but no reason new stuff can't use the new wording :).

I will make that change in all places. So is it compute Ignition or worker Ignition?

Unless the machineconfigpool is changing to compute I think we should leave this as is.

@sdodson is your comment against s/worker/compute/ or just the Ignition case?

just in relation to Ignition

wking · 2019-09-10T16:15:38Z

+### Goals
+
+As part of this enhancement we plan to do the following:
+* Prepare an already provisioned Windows worker node to join the cluster


Clarify the boundary between things the end-user does (just create a machine running Windows) and things Red Hat (tooling) does (install a kubelet, etc. etc.)? Currently that's all swept into "provision" and "Prepare". Maybe something like:

As part of this enhancement we plan to provide workflows for installing and upgrading OpenShift compute components (kubelet, OVN, and the Windows Machine Config Bootstrapper) on user-provided Windows machines. It will be up to the cluster administrator to initiate both installs and upgrades.

I will go with your suggestion.

wking · 2019-09-10T16:18:01Z

+
+The inputs that the WMCB requires are:
+* kubelet location on the local disk
+* Worker ignition location on the local disk


nit: s/ignition/Ignition/

wking · 2019-09-10T16:20:27Z

+We plan to have all the repositories associated with this effort fully
+integrated with Prow CI and run e2e tests for every PR that is opened. These
+e2e tests will involve bringing up a cluster on all supported cloud providers,
+instantiating a Windows node and running workloads on it. We also plan to add


This doesn't cover upgrade/downgrade testing. Probably call that out specifically and say whether or not you plan on covering it in CI.

We are not planning on covering upgrade/downgrade testing using CI in the 4.3 timeframe. I will mention that.

wking · 2019-09-10T16:23:10Z

+
+### Version Skew Strategy
+
+We plan to maintain kubelet major version parity with the Linux counterpart.


You'll have to have skew during upgrades across the boundary though. Maybe say something about having CI coverage when we get to that point so you can ensure that at least some Windows kubelets will successfully handle lagging by one major version behind the rest of the cluster. Or say that before a major bump folks might have to drain/wipe their Windows compute and re-attach them as fresh compute after the major bump?

We are planning on advising the customer to exactly what is done in the BYO RHEL scenario i.e. drain the Windows compute nodes using the Ansible playbook. I will add a line about that.

Initial enhancement proposal for productization of Windows Containers that will enable running of Windows workloads on an OpenShift cluster.

aravindhp · 2019-09-10T19:44:34Z

@wking I have addressed your comments. Please take a look.

ravisantoshgudimetla · 2019-09-18T00:46:06Z

@crawford - Can you LGTM the PR?

sdodson · 2019-09-18T02:07:15Z

/lgtm

openshift-ci-robot · 2019-09-18T02:07:27Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aravindhp, crawford, sdodson

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [aravindhp,sdodson]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

derekwaynecarr · 2019-09-26T16:33:18Z

+
+The actions that the WMCB will perform are:
+* Install / upgrade and configure the kubelet
+* Parse the worker Ignition and extract the bootstrap kubeconfig and the


kubelet configuration varies for linux versus windows hosts, this has to be a separate pool.

It's only used for the cluster coordinates and bootstrapping credentials.

English mistake: sums up => adds up

Updates for master -> main branch renamings

OCPEDGE-1458: Addressing post-architecture review feedback

- Enhance motivation section and shorten summary - Remove downgrade from goals, move day 0 installation to non-goals - Add goal to preserve user-facing Gateway API experience - Clarify upgrade migration, drift, and image sourcing wording - Add Hypershift and OKE notes - Update maintenance/testing burden in drawbacks - Remove "Loss of OLM Benefits" drawback - Move Istio CRD management question from openshift#5 to openshift#2 - Update feature gate and downgrade section wording

* NE-2470: Gateway API without OLM Adds enhancements/ingress/gateway-api-without-olm.md for installing Gateway API support without OLM dependency. This enhancement transitions cluster-ingress-operator from creating an OLM Subscription to installing istiod directly using Helm charts and sail-operator libraries. This eliminates the OLM dependency, avoids conflicts with existing OSSM subscriptions, enables Gateway API on clusters without OLM/Marketplace capabilities, and allows faster Gateway API releases independent of OLM release cycles. * NE-2470: Gateway API without OLM Address code review feedback and update enhancement for consistency. - Add platform feature language and layered product user story - Add goals for simplified testing/releases and lifecycle management - Add Component Versioning section and upgrade workflows - Add CRD mapping/reasoning for layered products - Add Resource Drift risk and webhook questions - Update test scenarios for upgrade validation - Fix wording inconsistencies throughout document * NE-2470: Gateway API without OLM Add pre-release testing details and custom image sources non-goal. - Add subsection on testing pre-releases of OSSM - Clarify custom image sources and mirroring are out of scope * NE-2470: Gateway API without OLM - Clarify workflow titles to be trigger-based on Gateway API installation state - Add Alternative 3: CIO manages sail-operator deployment directly - Update Open Questions with answers for CRD handling and webhook management - Add risk for future sail-operator feature requirements - Add drawback for pre-release testing workflow complexity - Update migration detection to only check for Istio CR - Add migration timeline note * NE-2470: Gateway API without OLM - Add PeerAuthentication to managed Istio CRDs for Kuadrant mTLS - Update CRD descriptions with RHOAI requirements - Add goals: reduce resource overhead and enable OKE support - Add alternative for using existing sail libraries without enhancements - Add risk about sail-operator library dependency coordination * NE-2470: Gateway API without OLM - Clarify CRDs are for north/south ingress use cases that work without service mesh - Remove PeerAuthentication (requires service mesh) - Add note that OSSM team will provide library functions for CRD management * NE-2470: Gateway API without OLM - Move CRD Management to Open Questions - Add Tech Preview feature gate graduation path for 4.22 * NE-2470: Gateway API without OLM - Add support procedures to satisfy markdownlint job - Replace vague "lifecycle management" with "reduce component dependencies" - Add note explaining "Helm-based" vs "OLM-based" terminology * Address Miciah's review comments - Enhance motivation section and shorten summary - Remove downgrade from goals, move day 0 installation to non-goals - Add goal to preserve user-facing Gateway API experience - Clarify upgrade migration, drift, and image sourcing wording - Add Hypershift and OKE notes - Update maintenance/testing burden in drawbacks - Remove "Loss of OLM Benefits" drawback - Move Istio CRD management question from #5 to #2 - Update feature gate and downgrade section wording

Address latest comments

Per Alberto's direction, KMS key validation should live in the storage stack, not in HCCO. The aws-ebs-csi-driver-operator in openshift/csi-operator already uses the AWS SDK v2 for volume tagging with IRSA credentials — a new EBSKMSKeyValidationController follows the same pattern. Changes: - Validation probe moves from HCCO to csi-operator's EBS driver package - Probe runs continuously (not day-1-only), resolving stale-condition concern - Condition chain: ClusterCSIDriver.status -> HCCO reads -> HCP -> HC - IAM permissions updated: add kms:Encrypt for validation (4 total) - Cross-repo dependency documented (openshift/csi-operator + openshift/hypershift) - Remove StorageClassState conflation (adversarial review finding openshift#2) - Fix stale condition issue (adversarial review finding openshift#5)

- Replace simple root-ownership check with full trusted-path rule: resolve symlinks, check every ancestor, final object, and directory contents for root ownership and no group/other write permission - Add Non-Goal openshift#5: protection against root users is out of scope - Add Non-Goal openshift#3 clarification: MicroShift validates ownership/permissions but not semantic contents - Extend validation section with symlink resolution rationale (ostree/bootc layouts depend on symlinks) - Add risk entry for TOCTOU with mitigation (root-only can modify root-owned) - Add Alternatives: "checking only final directory" and "rejecting symlinks" - Expand unit tests: ancestor checks, contents checks, symlink cases, injected stat function note - Expand integration tests: unsafe binary, unsafe config file, unsafe ancestor - Update Drawbacks to reference trusted-path rule instead of permissions check - Upgrade/Downgrade: explain why lenient fallback is safe for all pre-feature versions and add Y-1 upgrade test reference Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

openshift-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Sep 3, 2019

openshift-ci-robot requested review from joelanford and mfojtik September 3, 2019 17:44

openshift-ci-robot requested review from crawford, ravisantoshgudimetla and sdodson September 3, 2019 17:45

sdodson approved these changes Sep 6, 2019

View reviewed changes

Comment thread windows-containers/productization.md Outdated

Comment thread windows-containers/productization.md

Comment thread windows-containers/productization.md Outdated

Comment thread windows-containers/productization.md

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 6, 2019

aravindhp force-pushed the winc-productization branch 2 times, most recently from aa75653 to ae78830 Compare September 9, 2019 19:41

ravisantoshgudimetla reviewed Sep 10, 2019

View reviewed changes

Comment thread windows-containers/productization.md Outdated

aravindhp force-pushed the winc-productization branch from ae78830 to 2173dda Compare September 10, 2019 14:39

wking reviewed Sep 10, 2019

View reviewed changes

Windows Containers productization design

ebb0e03

Initial enhancement proposal for productization of Windows Containers that will enable running of Windows workloads on an OpenShift cluster.

aravindhp force-pushed the winc-productization branch from 2173dda to ebb0e03 Compare September 10, 2019 17:42

crawford approved these changes Sep 18, 2019

View reviewed changes

openshift-ci-robot assigned sdodson Sep 18, 2019

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Sep 18, 2019

openshift-merge-robot merged commit f796b93 into openshift:master Sep 18, 2019

derekwaynecarr reviewed Sep 26, 2019

View reviewed changes

retroflexer mentioned this pull request Mar 5, 2020

enhancements/etcd/disaster-recovery-with-ceo: add backup proposal #240

Closed

openshift-merge-robot pushed a commit that referenced this pull request Oct 18, 2021

Merge pull request #5 from jotak/adds-up

a1ea942

English mistake: sums up => adds up

cgwalters pushed a commit to cgwalters/enhancements that referenced this pull request Oct 19, 2021

Merge pull request openshift#5 from dustymabe/dusty-branch-renaming

8cf355d

Updates for master -> main branch renamings

jaypoulz pushed a commit to jaypoulz/enhancements that referenced this pull request Feb 26, 2025

Merge pull request openshift#5 from jaypoulz/tnf

4a2e899

OCPEDGE-1458: Addressing post-architecture review feedback

openshift-merge-bot Bot pushed a commit that referenced this pull request Mar 17, 2026

Merge pull request #5 from tmshort/latest-comments

a571863

Address latest comments

muraee mentioned this pull request Jul 3, 2026

CNTRLPLANE-3776: In-Place Config Refresh for NodePool Worker Nodes #2053

Open

2 tasks


		### Version Skew Strategy

		We plan to maintain kubelet major version parity with the Linux counterpart.

Uh oh!

Conversation

aravindhp commented Sep 3, 2019

Uh oh!

aravindhp commented Sep 3, 2019

Uh oh!

sdodson left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aravindhp commented Sep 10, 2019

Uh oh!

ravisantoshgudimetla commented Sep 18, 2019

Uh oh!

sdodson commented Sep 18, 2019

Uh oh!

openshift-ci-robot commented Sep 18, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants