Installer does not work with pre-provisioned vms

[yrobla]

Platform (aws|libvirt|openstack): libvirt

I am using the installer with libvirt, but not in the traditional way. Instead, i'm generating the ignition and manifests files first, and then i'm pre-creating the vms. I am passing those ignition files to the pre-created vms.
This procedure was working fine, until there was a major change on how etcd cluster is created. Since then it's failing for me, even if i use the latest images.
Issue seems to be with etcd cluster on master, it's never reaching to a healthy point. In bootstrap, i always can see :
043548de8fd5 quay.io/coreos/etcd:v3.2.14 /usr/local/bin/etcd... 3 minutes ago Up 2 minutes ago etcdctl
f49c5993cfec quay.io/coreos/kube-etcd-signer-server:678cc8e6841e2121ebfdb6e2db568fce290b67d6 kube-etcd-signer-se... 13 minutes ago Up 13 minutes ago etcd-signer

In etcd-signer i can see some errors:

2018/11/09 16:32:19 [INFO] POST/apis/certificates.k8s.io/v1beta1/certificatesigningrequests
ERROR: logging before flag.Parse: E1109 16:32:19.354366 1 signer.go:292] Error signing CSR provided in request from agent: error parsing profile: invalid organization

master just iterates, is not able to be registered, as it cannot contact the cluster.

I can show how am i working, it's a compose of different jobs:

http://10.8.125.26:8080/job/build-openshift-installer -> installs latest code, generates manifests and ignition files
http://10.8.125.26:8080/job/build-openshift-cluster -> creates the vms, embeds ignition files on bootstrap and master

Also logs with the failures:
http://10.8.125.26/logs/

bootstrap.log and master.log are the logs provided by the full openshift installer
bootstrap_custom.log and master_custom.logs are the logs provided by my installer with custom workflow

Also you can see how all the code is glued together on:
https://github.com/redhat-nfvpe/edge-poc-netlab

[crawford]

Can you attach those logs to this report? A gist or <Details> would be fine.

[yrobla]

master_custom.log
bootstrap_custom.log
master.log
bootstrap.log

[wking]

2018/11/09 16:32:19 [INFO] POST/apis/certificates.k8s.io/v1beta1/certificatesigningrequests
ERROR: logging before flag.Parse: E1109 16:32:19.354366 1 signer.go:292] Error signing CSR provided in request from agent: error parsing profile: invalid organization

This is discussed here, it should be temporary.

[yrobla]

Then it is a red herring.. but the problem with etcd cluster not being formed continues. What other issues can be causing it ?

[yrobla]

Following the troubleshooting guide, i executed that on master node:

sudo crictl logs $(sudo crictl ps --pod=$(sudo crictl pods --name=etcd-member --quiet) --quiet)

And i got this error:
E1110 09:18:21.842044 1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.testcluster.edge.testing on 192.168.126.1:53: no such host

Where does this address come from? Do i need to create some additional vm, and fix the ip address to that?

My network setup is something like:

<network connections='2'>
  <name>cluster</name>
  <uuid>d4a13614-be5e-4f4a-88f2-63c47d01c2a4</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='br0' stp='on' delay='0'/>
  <mac address='52:54:00:b7:97:26'/>
  <domain name='edge.testing' localOnly='yes'/>
  <dns>
    <srv service='etcd-server-ssl' protocol='tcp' domain='edge.testing' target='testcluster-etcd-0.edge.testing' port='2380' weight='10'/>
    <host ip='192.168.126.2'>
      <hostname>testcluster-api</hostname>
    </host>
    <host ip='192.168.126.3'>
      <hostname>testcluster-api</hostname>
      <hostname>testcluster-etcd-0</hostname>
    </host>
    <host ip='192.168.126.0'>
      <hostname>testcluster</hostname>
    </host>
  </dns>
  <ip family='ipv4' address='192.168.126.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.126.2' end='192.168.126.254'/>
      <host mac='52:54:00:cc:cc:01' name='testcluster-bootstrap' ip='192.168.126.2'/>
      <host mac='52:54:00:cc:cc:02' name='testcluster-master-0' ip='192.168.126.3'/>
      <host mac='52:54:00:cc:cc:03' name='testcluster-master-1' ip='192.168.126.4'/>
      <host mac='52:54:00:cc:cc:04' name='testcluster-master-2' ip='192.168.126.5'/>
      <bootp file='bootstrap.ipxe' server='192.168.126.1'/>
    </dhcp>
  </ip>
</network>

[wking]

E1110 09:18:21.842044 1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.testcluster.edge.testing on 192.168.126.1:53: no such host

Where does this address come from?

That's originally from #526. Looks like you may have missed #596 / #598, since you have domain='edge.testing instead of testcluster.edge.testing. Once you fix that, you can check with:

$ $ dig SRV "_etcd-server-ssl._tcp.${OPENSHIFT_INSTALL_CLUSTER_NAME}.${OPENSHIFT_INSTALL_BASE_DOMAIN}" +short
0 10 2380 wking-etcd-0.installer.testing.

(or whatever response is appropriate for your cluster).

[yrobla]

You are right, i was missing the cluster name from the entry. Adding that solved the problem, thx!