Kubernetes API is not available after install using latest dev-scripts

[chunfuwen]

install console log

---

level=debug msg="Still waiting for the Kubernetes API: Get https://api.ostest.test.metalkube.org:6443/version?timeout=32s: dial tcp 192.168.111.5:6443: connect: connection refused"
level=debug msg="Still waiting for the Kubernetes API: Get https://api.ostest.test.metalkube.org:6443/version?timeout=32s: dial tcp 192.168.111.5:6443: connect: connection refused"
level=debug msg="Fetching \"Install Config\"..."
level=debug msg="Loading \"Install Config\"..."
level=debug msg="  Loading \"SSH Key\"..."
level=debug msg="  Loading \"Base Domain\"..."
level=debug msg="    Loading \"Platform\"..."
level=debug msg="  Loading \"Cluster Name\"..."
level=debug msg="    Loading \"Base Domain\"..."
level=debug msg="  Loading \"Pull Secret\"..."
level=debug msg="  Loading \"Platform\"..."
level=debug msg="Using \"Install Config\" loaded from state file"
level=debug msg="Reusing previously-fetched \"Install Config\""
level=error msg="Cannot fetch the bootstrap and control plane host addresses from state file for baremetal platform"
level=error msg="bootstrap host address and at least one control plane host address must be provided"
level=fatal msg="waiting for Kubernetes API: context deadline exceeded"
make: *** [ocp_run] Error 1

---

master node log for etcd

[core@master-0 ~]$ sudo crictl logs $(sudo crictl ps --pod=$(sudo crictl pods --name=etcd-member --quiet) --quiet)
+ source /run/etcd/environment
++ export ETCD_DISCOVERY_SRV=ostest.test.metalkube.org
++ ETCD_DISCOVERY_SRV=ostest.test.metalkube.org
++ ETCD_IPV4_ADDRESS=192.168.111.20
++ ETCD_DNS_NAME=etcd-0.ostest.test.metalkube.org
++ ETCD_WILDCARD_DNS_NAME='*.ostest.test.metalkube.org'
+ '[' -e /etc/ssl/etcd/system:etcd-server:etcd-0.ostest.test.metalkube.org.crt -a -e /etc/ssl/etcd/system:etcd-server:etcd-0.ostest.test.metalkube.org.key ']'
+ kube-client-agent request --kubeconfig=/etc/kubernetes/kubeconfig --orgname=system:etcd-servers --assetsdir=/etc/ssl/etcd '--dnsnames=localhost,etcd.kube-system.svc,etcd.kube-system.svc.cluster.local,etcd.openshift-etcd.svc,etcd.openshift-etcd.svc.cluster.local,*.ostest.test.metalkube.org' --commonname=system:etcd-server:etcd-0.ostest.test.metalkube.org --ipaddrs=192.168.111.20,127.0.0.1
ERROR: logging before flag.Parse: E0827 08:29:15.987360       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:29:25.986507       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:29:45.988025       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: net/http: TLS handshake timeout
ERROR: logging before flag.Parse: E0827 08:29:56.037335       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: read tcp 192.168.111.20:49506->192.168.111.5:6443: read: connection reset by peer
ERROR: logging before flag.Parse: E0827 08:30:05.986309       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:30:15.986797       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:30:25.986735       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:30:35.986845       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:30:55.987163       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: net/http: TLS handshake timeout
ERROR: logging before flag.Parse: E0827 08:31:05.987512       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:31:15.986275       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:31:25.986536       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:31:35.986467       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:31:45.986553       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:31:55.986145       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:32:05.986902       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:32:15.986221       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:32:25.986357       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:32:45.986733       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: net/http: TLS handshake timeout
ERROR: logging before flag.Parse: E0827 08:32:55.986656       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:33:05.985978       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:33:15.986121       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:33:25.986330       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:33:35.986483       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:33:45.986863       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:33:55.986413       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:34:05.986601       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:34:15.986423       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:34:25.986429       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:34:35.986807       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:34:45.986458       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:34:55.986482       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:35:05.986371       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:35:15.987388       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:35:25.986737       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:35:45.986822       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: net/http: TLS handshake timeout

[russellb]

Can you get the bootkube log from the bootstrap VM?

[chunfuwen]

boostrap vm log

---

[chwen@dell-r730-058 dev-scripts]$ ./bootstrap_openshift_log.sh 
Attempting to follow openshift.service on 192.168.111.56 ...
Warning: Permanently added '192.168.111.56' (ECDSA) to the list of known hosts.
-- Logs begin at Tue 2019-08-27 08:14:05 UTC. --
Aug 28 00:54:12 localhost openshift.sh[1675]: error: unable to recognize "./90_metal3_baremetalhost_crd.yaml": Get https://localhost:6443/api?timeout=32s: dial tcp [::1]:6443: connect: connection refused
Aug 28 00:54:12 localhost openshift.sh[1675]: kubectl create --filename ./90_metal3_baremetalhost_crd.yaml failed. Retrying in 5 seconds...
Aug 28 00:54:17 localhost openshift.sh[1675]: error: unable to recognize "./90_metal3_baremetalhost_crd.yaml": Get https://localhost:6443/api?timeout=32s: dial tcp [::1]:6443: connect: connection refused
Aug 28 00:54:17 localhost openshift.sh[1675]: kubectl create --filename ./90_metal3_baremetalhost_crd.yaml failed. Retrying in 5 seconds...
Aug 28 00:54:22 localhost openshift.sh[1675]: error: unable to recognize "./90_metal3_baremetalhost_crd.yaml": Get https://localhost:6443/api?timeout=32s: dial tcp [::1]:6443: connect: connection refused
Aug 28 00:54:22 localhost openshift.sh[1675]: kubectl create --filename ./90_metal3_baremetalhost_crd.yaml failed. Retrying in 5 seconds...
Aug 28 00:54:27 localhost openshift.sh[1675]: error: unable to recognize "./90_metal3_baremetalhost_crd.yaml": Get https://localhost:6443/api?timeout=32s: dial tcp [::1]:6443: connect: connection refused
Aug 28 00:54:27 localhost openshift.sh[1675]: kubectl create --filename ./90_metal3_baremetalhost_crd.yaml failed. Retrying in 5 seconds...
Aug 28 00:54:32 localhost openshift.sh[1675]: error: unable to recognize "./90_metal3_baremetalhost_crd.yaml": Get https://localhost:6443/api?timeout=32s: dial tcp [::1]:6443: connect: connection refused
Aug 28 00:54:32 localhost openshift.sh[1675]: kubectl create --filename ./90_metal3_baremetalhost_crd.yaml failed. Retrying in 5 seconds...
Aug 28 00:54:37 localhost openshift.sh[1675]: error: unable to recognize "./90_metal3_baremetalhost_crd.yaml": Get https://localhost:6443/api?timeout=32s: dial tcp [::1]:6443: connect: connection refused
Aug 28 00:54:37 localhost openshift.sh[1675]: kubectl create --filename ./90_metal3_baremetalhost_crd.yaml failed. Retrying in 5 seconds...
Aug 28 00:54:43 localhost openshift.sh[1675]: error: unable to recognize "./90_metal3_baremetalhost_crd.yaml": Get https://localhost:6443/api?timeout=32s: dial tcp [::1]:6443: connect: connection refused
Aug 28 00:54:43 localhost openshift.sh[1675]: kubectl create --filename ./90_metal3_baremetalhost_crd.yaml failed. Retrying in 5 seconds...
Aug 28 00:54:48 localhost openshift.sh[1675]: error: unable to recognize "./90_metal3_baremetalhost_crd.yaml": Get https://localhost:6443/api?timeout=32s: dial tcp [::1]:6443: connect: connection refused
Aug 28 00:54:48 localhost openshift.sh[1675]: kubectl create --filename ./90_metal3_baremetalhost_crd.yaml failed. Retrying in 5 seconds...
Aug 28 00:54:53 localhost openshift.sh[1675]: error: unable to recognize "./90_metal3_baremetalhost_crd.yaml": Get https://localhost:6443/api?timeout=32s: dial tcp [::1]:6443: connect: connection refused
Aug 28 00:54:53 localhost openshift.sh[1675]: kubectl create --filename ./90_metal3_baremetalhost_crd.yaml failed. Retrying in 5 seconds...
Aug 28 00:54:58 localhost openshift.sh[1675]: error: unable to recognize "./90_metal3_baremetalhost_crd.yaml": Get https://localhost:6443/api?timeout=32s: dial tcp [::1]:6443: connect: connection refused
Aug 28 00:54:58 localhost openshift.sh[1675]: kubectl create --filename ./90_metal3_baremetalhost_crd.yaml failed. Retrying in 5 seconds..

[stbenjam]

You're most likely hitting the issue we found with podman on the current verison of rhcos. podman isn't returning exit codes correctly, so when etcd fails to start the first time, bootkube doesn't retry. We need openshift/installer to update their rhcos release, and then merge #766

[hardys]

Closed via #766