"openshift-install destroy cluster" leaves auth dir, breaking next install

[sttts]

After installing, there is the auth/kubeconfig file in the current checkout. When destroying the cluster, this file is not deleted. A following cluster install load this files, but it does not match the new certificates. Hence, the cluster does not work as expected for the user.

[sttts]

Couldn't double check this due to the current AWS quota issues. Please confirm if this is an issue.

[wking]

A following cluster install load this files...

More on this over here and later. For now, the easiest approach is probably to use --dir=whatever and then after you destroy the cluster rm -rf whatever to give you a clean slate for the next run.

[crawford]

I don't follow how the presence of auth/kubeconfig caused the certificate mismatch. If the installer's state file was still present, it would have reused all of the previously generated TLS assets, resulting in valid certificates. Did you remove the installer's state file but leave the kubeconfig?

[sttts]

Did you remove the installer's state file but leave the kubeconfig?

Yes, I did.

[stlaz]

Encountered the very same problem - several times. Removing the auth/ directory always fixes the cert mismatch.

[crawford]

Yes, that makes sense. You are effectively telling the installer to ignore whatever kubeconfig it has generated and to instead use the one you have provided (which will almost always be incorrect).

We should revisit this UX issue soon.

[mazzystr]

I have been doing the following...

• bin/openshift-install destroy-cluster
• rm -rf terraform.state terraform.tfvars terraform.tfstate .openshift_install_state.json metadata.json
• bin/openshift-install cluster
• mv auth/kubeconfig ~/.kube/ccallegar
• export KUBECONFIG=$HOME/.kube/ccallegar

The result is a failed cluster. All instances behind the int and ext aws elb are OutOfService. Kublet services are failed on instances ... essentially there is no OpenShift cluster.

You will see the following output (and systemd journal logs) when starting kublet ....

[root@ip-10-0-7-17 ~]# /usr/bin/hyperkube kubelet --config=/etc/kubernetes/kubelet.conf --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig --rotate-certificates --kubeconfig=/var/lib/kubelet/kubeconfig --container-runtime=remote --container-runtime-endpoint=/var/run/crio/crio.sock --allow-privileged --node-labels=node-role.kubernetes.io/master --minimum-container-ttl-duration=6m0s --client-ca-file=/etc/kubernetes/ca.crt --cloud-provider=aws --anonymous-auth=false --register-with-taints=node-role.kubernetes.io/master=:NoSchedule
Flag --rotate-certificates has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --allow-privileged has been deprecated, will be removed in a future version
Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Flag --client-ca-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --anonymous-auth has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --rotate-certificates has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --allow-privileged has been deprecated, will be removed in a future version
Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Flag --client-ca-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --anonymous-auth has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
I1030 18:27:51.651642 4254 server.go:418] Version: v1.11.0+d4cacc0
I1030 18:27:51.652086 4254 aws.go:1032] Building AWS cloudprovider
I1030 18:27:51.652953 4254 aws.go:994] Zone not specified in configuration file; querying AWS metadata service
I1030 18:27:51.836182 4254 tags.go:76] AWS cloud filtering on ClusterID: ccallegar
F1030 18:27:51.842709 4254 server.go:262] failed to run Kubelet: cannot create certificate signing request: Post https://ccallegar-api.ccallegar.sysdeseng.com:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: EOF

Executing the following fixes everything!

• bin/openshift-install destroy-cluster
• rm -rf terraform.state terraform.tfvars terraform.tfstate .openshift_install_state.json metadata.json auth
• bin/openshift-install cluster
• mv auth/kubeconfig ~/.kube/ccallegar
• export KUBECONFIG=$HOME/.kube/ccallegar

[wking]

rm -rf terraform.state terraform.tfvars terraform.tfstate .openshift_install_state.json metadata.json auth

Docs for the easier --dir approach are in flight with #532.

[sttts]

I still think the UX is broken. There should be a default for --dir and a warning or even error if that dir already exists, with an override if it is really intended to re-use a dir.

[wking]

There should be a default for --dir...

This would make life easier for devs, where cluster installs happen many times a day. But I think it would make life slightly more complicated for external users, who are more likely to successfully launch a single cluster and then walk away from the installer for weeks+. I'm fine biasing the UX in favor of external users.

However, folks who preserve their state file (or, with #556, their state directory) should be fine reusing an earlier run's assets exept for the 30-minute-validity kubelet-client cert and its descendants. I think we could guard against that with asset-load-time validators.

The only remaining issue then would be folks who removed parents but not frozen child assets between runs. #556 does a better job tracking frozen/modified assets, with this warning (which we can strengthen as discussed there) in the dangerous case.

[jeremyeder]

I wanted to post an error message from a state my system is in now which I think is related to this issue. openshift-install cluster delete fails with the following message:

DependencyViolation: resource sg-0d577e26d576aabc9 has a dependent object