MachineConfigs can be garbage collected while a node is still booting

[cgwalters]

Splitting this out of #273 (comment)
#273 (comment)

Here are relevant mcc logs from a cluster spun up by that PR:

I0114 22:26:10.482441       1 render_controller.go:437] Generated machineconfig master-46c05bfb9cb3d4e05608277bb2cb0a5d from 1 configs: [{MachineConfig  00-master  machineconfiguration.openshift.io/v1  }]
I0114 22:26:11.589272       1 render_controller.go:437] Generated machineconfig master-7734c782bad1ead0f8ef5b6affcaf35c from 2 configs: [{MachineConfig  00-master  machineconfiguration.openshift.io/v1  } {Machin
eConfig  00-master-osimageurl  machineconfiguration.openshift.io/v1  }]
I0114 22:26:12.484212       1 render_controller.go:437] Generated machineconfig master-383ca913310d861fee0be89e6f1d0127 from 3 configs: [{MachineConfig  00-master  machineconfiguration.openshift.io/v1  } {Machin
eConfig  00-master-osimageurl  machineconfiguration.openshift.io/v1  } {MachineConfig  00-master-ssh  machineconfiguration.openshift.io/v1  }]
I0114 22:26:14.483810       1 render_controller.go:437] Generated machineconfig master-a206c9459a44d859587164a68bb484f2 from 4 configs: [{MachineConfig  00-master  machineconfiguration.openshift.io/v1  } {Machin
eConfig  00-master-osimageurl  machineconfiguration.openshift.io/v1  } {MachineConfig  00-master-ssh  machineconfiguration.openshift.io/v1  } {MachineConfig  01-master-kubelet  machineconfiguration.openshift.io/
v1  }]

You can see that this caused very fast churn in the machineconfigs, and the previous ones were GCd. But - there were secondary masters that were still booting and expecting to be able to find that MC.
This is a tricky problem - we need to have a way to avoid pruning "in flight" MCs passed from the MCC to Ignition.

[cgwalters]

This problem is actually fairly new (as of basically since Friday) because of the near-simultaneous addition of -ssh and -kubelet MCs which causes MC regeneration churn immediately on cluster bringup, and I'm currently going to add another.

[ashcrow]

/kind bug

[kikisdeliveryservice]

As a data point, the -ssh mc was added last Monday, so any idea why you saw the issues begin on Friday? I also haven't see any MC race symptoms running on AWS.

[cgwalters]

As a data point, the -ssh mc was added last Monday, so any idea why you saw the issues begin on Friday?

Sorry, I was wrong. It's likely the -kubelet one on top of the -ssh - but both are pretty recent.

[jlebon]

Hmm, wonder if it'd work to add another owner reference from the MCD on the MC before handling it. Let me look into that.

[jlebon]

So, just to do a brain dump on this. I'm still trying to reproduce this locally. It doesn't trigger at least when I manually create MCs in rapid succession (this is with #303 reverted). One thing that caught my eye though is that there is no code today that actually deletes MachineConfigs, right?

The render controller does try to remove the pool ownerReference from previously generated MachineConfigs (that actually doesn't work right now; working on a patch to fix that), but even so, deleting the ownerRef doesn't automatically delete the object, it just orphans them (which is actually an issue; we need to figure out garbage collection).

[cgwalters]

The render controller does try to remove the pool ownerReference from previously generated MachineConfigs (that actually doesn't work right now; working on a patch to fix that),

Something in that CI run was pruning MCs though.

it just orphans them (which is actually an issue; we need to figure out garbage collection).

Doesn't Kube itself do the GC https://blog.openshift.com/garbage-collection-custom-resources-available-kubernetes-1-8/ ?

We should indeed figure this issue out better but it's not in the blocking path right now I'd say.

[ashcrow]

We should indeed figure this issue out better but it's not in the blocking path right now I'd say.

Agreed