Move layered node image definition to new openshift/node repo

[jlebon]

Currently, this repo contains both the definition of the base RHCOS bootable container (which includes image.yaml, used for disk image creation) and the layered node image (Containerfile).

Those two are attached to different versioning. E.g. the layered node image content should keep following OCP branching, but that doesn't make sense for the RHEL-versioned base image. Additionally, having both defined in the same repo is confusing because it's not clear which file pertains to which and what exactly CI is testing.

See also: https://github.com/openshift/enhancements/blob/master/enhancements/rhcos/split-rhcos-into-layers.md#re-organize-repos

Here's a starting proposal:

• We create a new openshift/node repo and move everything related to building the node image there. We can then also start adding more OCP-specific things, like some of the MCO static templates. That repo follows OCP branching.
• We stop branching this repo for OCP versions. We start branching this repo for RHEL EUS versions. Once dual stream happens, one branch would capture definitions for both RHEL 9.x and RHEL 10.y versions that were released together on the OCP side.
• We rename this repo to openshift/coreos? Optional, but I think it'd be clearer than the generic "os" name.

One thing I had considered was to have the RHCOS/SCOS base definition be defined in Gitlab instead, closer to rhel-bootc/centos-bootc. But I think the pull from OpenShift CI and its integration with GitHub is too strong to ignore. A lot of CI happens at the node layer, but I want us to also be able to easily run cluster tests when the base layer changes.

[dustymabe]

I'm generally in favor, but do have some comments.

• We create a new openshift/node repo and move everything related to building the node image there. We can then also start adding more OCP-specific things, like some of the MCO static templates. That repo follows OCP branching.

SGTM

• We stop branching this repo for OCP versions. We start branching this repo for RHEL EUS versions. Once dual stream happens, one branch would capture definitions for both RHEL 9.x and RHEL 10.y versions that were released together on the OCP side.

This one is less clear. We branch for RHEL EUS versions but also define el9 and el10 (and eventually el11) together in the latest development branch?

• We rename this repo to openshift/coreos? Optional, but I think it'd be clearer than the generic "os" name.

meh.. maybe, but I would put this low on the priority list especially if there is a lot of work that has to be done in various places to support it.

[jlebon]

• We stop branching this repo for OCP versions. We start branching this repo for RHEL EUS versions. Once dual stream happens, one branch would capture definitions for both RHEL 9.x and RHEL 10.y versions that were released together on the OCP side.

This one is less clear. We branch for RHEL EUS versions but also define el9 and el10 (and eventually el11) together in the latest development branch?

I think another way to say this is that we follow RHEL branching. Basically the main branch tracks e.g. c9s/9.6 and c10s/10.0 and then when RHEL branches for those, we also branch and create a new e.g. rhel-9.6-10.0 branch.

• We rename this repo to openshift/coreos? Optional, but I think it'd be clearer than the generic "os" name.

meh.. maybe, but I would put this low on the priority list especially if there is a lot of work that has to be done in various places to support it.

I think if we want to do it, it would be better to do it at roughly the same time we do the node image move since that'll require some adaptation anyway in various places. In practice, a lot of things should Just Work I think thanks to GitHub's auto-redirect, though I'm sure that doesn't catch everything.

[dustymabe]

I think another way to say this is that we follow RHEL branching. Basically the main branch tracks e.g. c9s/9.6 and c10s/10.0 and then when RHEL branches for those, we also branch and create a new e.g. rhel-9.6-10.0 branch.

I think I'd be more in favor of having the branches be scoped specifically to a single release of RHEL. i.e. We'd make a rhel-9.6 AND a rhel-10.0 branch, versus a single one. Then changes are better scoped?

[dustymabe]

I think if we want to do it, it would be better to do it at roughly the same time

I see what you are saying with that argument and I agree. I still think I'd vote against a name change here since the work/value ratio is low IMO.

[jlebon]

I think I'd be more in favor of having the branches be scoped specifically to a single release of RHEL. i.e. We'd make a rhel-9.6 AND a rhel-10.0 branch, versus a single one. Then changes are better scoped?

Yeah, maybe. I see the appeal, but OTOH I think there's value also in thinking of the two X.Y releases as bound based on how we shipped it in OCP. Concretely, that's how the main branch will work and so it feels natural to branch that way too. It also makes backports that affect both easier to handle. Anyway, this is clearly something that we could change strategy on as we gain experience with dual stream.

[jlebon]

We rename this repo to openshift/coreos? Optional, but I think it'd be clearer than the generic "os" name.

OK, now thinking actually openshift/coreos-base maybe? Mostly, I think it'd be good if the naming reflects better the relationship between the two repos and with node and os, that's not at all clear.

[jlebon]

OK, https://github.com/coreos/rhel-coreos-config is now a thing. The main branch there is currently matched with the master branch of openshift/os.

Proposal

So the concrete proposal for immediate next steps are:

• Turn off 4.19-9.6 stream in pipeline.
• Move pipecfg to build the el-only variants (currently, rhel-9.6, c9s, and c10s) from https://github.com/coreos/rhel-coreos-config.
• Hook up f-c-c submodule bumps to go to https://github.com/coreos/rhel-coreos-config instead → workflows: bump coreos/rhel-coreos-config submodule coreos/fedora-coreos-config#3496
• Move openshift/release test definitions related to cosa building to target https://github.com/coreos/rhel-coreos-config instead. Keep test definitions related to node layering to target https://github.com/openshift/os. → config: add coreos/rhel-coreos-config release#64590
• Open PR to https://github.com/openshift/os to remove base compose files → NO-JIRA: tree: delete base compose files, rework README #1811
• Open PR to https://github.com/coreos/rhel-coreos-config to remove layered node image files. → NO-JIRA: tree: delete all files related to layered node image, delete ocp-rhel-9.6, update README coreos/rhel-coreos-config#2

FAQ

Why do this? Why not leave them together?

1. The node layer definition needs to be able to branch with OCP. The CoreOS layer definition cannot. We could say "keep maintaining the CoreOS layer in master", but apart from being confusing, we likely do want to be able to branch the CoreOS layer as well in the future, but under a cadence more in line with RHEL, not OCP.
2. It's confusing to have both the base CoreOS layer and the node layer be defined alongside each other. It also means that anything watching git for only one of those two layers will churn more often than necessary.

Why name it "rhel-coreos-config"? We would also build SCOS from that repo.

This is true. But:

1. We are building SCOS from that repo for now because it's a convenient place to build it. If the OKD team/community had the infrastructure and wanted to help co-maintain this, it would make sense to have that built in a separate repo (e.g. under the https://github.com/CentOS/ org). The primary artifact of this repo is still RHCOS.
2. The naming follows the pattern of https://github.com/coreos/fedora-coreos-config. There isn't a good codeword that encompasses both RHCOS and SCOS. Other suggestions included "el-coreos-config", "enterprise-coreos-config", or simply "coreos-config".

Open to more bikeshedding on the naming.

Why not put it in GitLab? That's where rhel-bootc and centos-bootc are defined.

This was considered. But Prow integration currently is GitHub-only, and I want it to be easy to e.g. launch a cluster from a PR to the rhel-coreos-config repo. (Though note this is currently not feasible, because we can't do a base compose for Prow purposes -- see also this related PR -- but it would be once we were a layered build on top of rhel-bootc. (Somewhat... we wouldn't be able to do a custom base image rebuild, because that's still rpm-ostree compose currently, but we'd be able to apply the layered bits on top.))

Why not put it under the https://github.com/openshift/ org? RHCOS/SCOS is only used by OpenShift/OKD.

This is true. But having it colocated with https://github.com/coreos/fedora-coreos-config makes sense too, both conceptually and maintenance-wise. It also emphasizes the fact that the RHCOS base layer has no OpenShift components in itself. Even if RHCOS is currently only used by OpenShift, it's helpful to think of it as clearly separating the "OS bits" from the "k8s bits", and "OS bits" fits better in the https://github.com/coreos/ org.

What are the downsides of doing this?

• More repos to watch/maintain.
• Changes that cut across both the base and the node layer can no longer be done atomically. Though note this is already somewhat true today. You need to wait until the base is rebuilt before you can build the node layer. And CI currently doesn't build both the base and the node layer in the same path.

[dustymabe]

+1 to the proposal.

[travier]

+1 from me. Thanks for the very detailed FAQ :)

[jlebon]

OK, this now needs coreos/fedora-coreos-pipeline#1142.

[jlebon]

OK, the remaining PRs here are:

• config: add coreos/rhel-coreos-config release#64590
• NO-JIRA: tree: delete base compose files, rework README #1811
• NO-JIRA: tree: delete all files related to layered node image, delete ocp-rhel-9.6, update README coreos/rhel-coreos-config#2

I think once those are merged, we can close this ticket.