Teach installer to specify OS version through oscontainer image URLs

[jlebon]

As mentioned in #267, the latest public RHCOS AMI will likely not be the latest RHCOS we want. Right now, MachineConfigs hardcode ://dummy as the OSImageURL: https://github.com/openshift/machine-config-operator/blob/a91ab5279755f87f0953f294e9add7584761a489/pkg/controller/template/render.go#L208. This should instead be the URL to the oscontainer image the installer would like to have installed. In more concrete terms: we could e.g. hardcode it for now as is currently done in ami.go for AMIs until we're ready to just always pick the latest, but also have something analogous to ec2AMIOverride, e.g. osImageURLOverride?

The MCD will then immediately update the node to the desired oscontainer upon startup.

[wking]

As mentioned in #267, the latest public RHCOS AMI will likely not be the latest RHCOS we want

Is this AMI-specific? Why would the AMI lag, but the libvirt image be current?

This should instead be the URL to the oscontainer image the installer would like to have installed.

This would effectively give users two knobs for the same thing:

• The AMI (or libvirt image, etc.?)
• A URL for the target OSTree (assuming an OSTree-capabale AMI)

Do we need both of those knobs? If AMI lag isn't too great, it seems like we could stick with just the AMI as the knob, and let post-install tooling like the machine API handle subsequent modifications. If the AMI is just an OSTree-capable stub (i.e. without some packages needed to run the installer/cluster), then can we just remove the AMI/libvirt-image knob completely and automatically pull an OSTree-capable stub without giving the user a choice (they could still choose their target OSTree)?

How does this all fit in with release tracks (tested, etc.) and automatic upgrades? Can the installer continue to ignore those (and leave them to other operators to handle or not)?

[jlebon]

Is this AMI-specific? Why would the AMI lag, but the libvirt image be current?

I think the long term plan is that only some (regularly updated, but "outdated") snapshots of RHCOS AMIs will be made public. So the installer needs to be able to work from a public RHCOS AMI and bring it up to date. Libvirt images are available internally just for development. (And I don't doubt we'll keep pushing out up to date private RHCOS AMIs as well).

Do we need both of those knobs? If AMI lag isn't too great, it seems like we could stick with just the AMI as the knob, and let post-install tooling like the machine API handle subsequent modifications.

Yeah, the AMI should be fully capable of joining the cluster seeing as it's just an older version of RHCOS. I guess the way I was thinking about it is that we should care less about the AMI used to bootstrap the cluster and more about the oscontainer we want to install since that's actually what will run the cluster. It seems like that setting should belong in examples/aws.yaml, right?

/cc @ashcrow to keep me honest

[jlebon]

Related issue on the OS side: openshift/os#307

[cgwalters]

I think big picture we want to always pivot early on in the installation.

I am working on redoing the os build process to use pivot.

[wking]

I think big picture we want to always pivot early on in the installation.

Ok. So are folks comfortable removing the AMI knob (and the libvirt image knob?) and replacing it (them) with oscontainer image URLs? The installer would just pick something sane for the initial image, and callers wanting a specific eventual OS would need to use override the default oscontainer image URL.

I guess you could point that at a local (caching) registry if you wanted to efficiently launch multiple nodes. And once the cluster comes up, maybe the machine-config operator could point new nodes at image streams in the cluster's own registry.

[cgwalters]

Ok. So are folks comfortable removing the AMI knob (and the libvirt image knob?)

Yes, though we will have changes that require updating those (in fact the recent partitioning switch is one) but those should become infrequent - let's say we limit ourselves to only respinning the AMI once every 6 months or so.

[ashcrow]

/cc @mrguitar

[jlebon]

So are folks comfortable removing the AMI knob (and the libvirt image knob?) and replacing it (them) with oscontainer image URLs?

Yeah, that sounds great!

So just to be explicit here, do we agree that the installer should take care of the initial pivot? It seems like leaving it up to the MCO would be more disruptive to the cluster than doing it upfront when not much is installed yet?

[ashcrow]

I think doing it upfront would be ideal. I wouldn't make it a hard requirement though.

[wking]

So just to be explicit here, do we agree that the installer should take care of the initial pivot?

Yes, although since #119 we are no longer handling workers in the installer, so this would just be for masters (and bootstrap?).

It seems like leaving it up to the MCO would be more disruptive to the cluster than doing it upfront when not much is installed yet?

I'm also concerned about agility without an AMI knob. A really old AMI may not be capable of running a modern master. If the installer pivots masters early, we don't have to worry about that.

[jlebon]

Yes, although since #119 we are no longer handling workers in the installer, so this would just be for masters (and bootstrap?).

Hmm, I'm not very familiar with the k8s cluster API, though are there hooks somewhere where we could pivot the workers before going any further?