Nodes status is 'registering' indefinitely in 12_csr_hack.sh

[e-minguez]

Describe the bug
Running the deployment in a baremetal environment, 12_csr_hack.sh doesn't finish. Nodes are registering forever:

+ echo -n 'Waiting for openshift-master-0 to stabilize ... '
Waiting for openshift-master-0 to stabilize ... + time_diff=-1567080201
+ [[ -1567080201 -gt '' ]]
++ curl -s -X GET http://localhost:8001/apis/metal3.io/v1alpha1/namespaces/openshift-machine-api/baremetalhosts/openshift-master-0/status -H 'Accept: application/json' -H 'Content-Type: application/json' -H 'User-Agent: link-machine-and-node'
++ jq .status.provisioning.state
++ sed 's/"//g'
+ state=registering
+ echo registering
registering
+ '[' registering = 'externally provisioned' ']'
+ sleep 5

$ oc get nodes
NAME                                         STATUS   ROLES           AGE   VERSION
kni1-master-0.example.com   Ready    master,worker   38m   v1.14.0+61fc89fab
kni1-master-1.example.com   Ready    master,worker   39m   v1.14.0+61fc89fab
kni1-master-2.example.com   Ready    master,worker   38m   v1.14.0+61fc89fab

$ oc get baremetalhosts -A
NAMESPACE               NAME                 STATUS   PROVISIONING STATUS   CONSUMER        BMC                                                      HARDWARE PROFILE   ONLINE   ERROR                                                       
openshift-machine-api   openshift-master-0   OK       registering           kni1-master-0   ipmi://r730xd-c13u23-drac.example.com                      true                                                                 
openshift-machine-api   openshift-master-1   OK       registering           kni1-master-1   ipmi://r730xd-c13u25-drac.example.com                      true                                                                 
openshift-machine-api   openshift-master-2   OK       registering           kni1-master-2   ipmi://r730xd-c13u27-drac.example.com                      true                                                                 

$ oc get machines -A
NAMESPACE               NAME            STATE   TYPE   REGION   ZONE   AGE
openshift-machine-api   kni1-master-0                                  37m
openshift-machine-api   kni1-master-1                                  37m
openshift-machine-api   kni1-master-2                                  37m

To Reproduce
commit 096f691

Expected/observed behavior
The script either finish by timeout or successfully.

Additional context

$ oc logs metal3-7685f8bdc9-cz5js -c metal3-baremetal-operator
{"level":"info","ts":1567081072.786168,"logger":"baremetalhost","msg":"Reconciling BareMetalHost","Request.Namespace":"openshift-machine-api","Request.Name":"openshift-master-2"}
{"level":"info","ts":1567081072.7863317,"logger":"baremetalhost","msg":"registering and validating access to management controller","Request.Namespace":"openshift-machine-api","Request.Name":"openshift-master-2","provisioningState":"registering","credentials":{"credentials":{"name":"openshift-master-2-bmc-secret","namespace":"openshift-machine-api"},"credentialsVersion":"15916"}}
{"level":"info","ts":1567081072.7863574,"logger":"baremetalhost_ironic","msg":"validating management access","host":"openshift-master-2"}
{"level":"info","ts":1567081072.8148258,"logger":"baremetalhost_ironic","msg":"found existing node by ID","host":"openshift-master-2"}
{"level":"info","ts":1567081072.8148444,"logger":"baremetalhost_ironic","msg":"current provision state","host":"openshift-master-2","lastError":"Error while attempting to adopt node 93652777-ffd4-422c-bcca-a4c0dabc8612: Cannot validate image information for node 93652777-ffd4-422c-bcca-a4c0dabc8612 because one or more parameters are missing from its instance_info and insufficent information is present to boot from a remote volume. Missing are: ['image_source', 'kernel', 'ramdisk'].","current":"adopt failed","target":"active"}
{"level":"info","ts":1567081072.8148665,"logger":"baremetalhost","msg":"response from validate","Request.Namespace":"openshift-machine-api","Request.Name":"openshift-master-2","provisioningState":"registering","provResult":{"Dirty":true,"RequeueAfter":0,"ErrorMessage":""}}
{"level":"info","ts":1567081072.8148785,"logger":"baremetalhost","msg":"host not ready","Request.Namespace":"openshift-machine-api","Request.Name":"openshift-master-2","provisioningState":"registering","wait":0}
{"level":"info","ts":1567081072.8148844,"logger":"baremetalhost","msg":"saving host status","Request.Namespace":"openshift-machine-api","Request.Name":"openshift-master-2","provisioningState":"registering","operational status":"OK","provisioning state":"registering"}
{"level":"info","ts":1567081072.8197143,"logger":"baremetalhost","msg":"done","Request.Namespace":"openshift-machine-api","Request.Name":"openshift-master-2","provisioningState":"registering","requeue":true,"after":0}
{"level":"info","ts":1567081072.8197699,"logger":"baremetalhost","msg":"Reconciling BareMetalHost","Request.Namespace":"openshift-machine-api","Request.Name":"openshift-master-2"}
{"level":"info","ts":1567081072.8198357,"logger":"baremetalhost","msg":"registering and validating access to management controller","Request.Namespace":"openshift-machine-api","Request.Name":"openshift-master-2","provisioningState":"registering","credentials":{"credentials":{"name":"openshift-master-2-bmc-secret","namespace":"openshift-machine-api"},"credentialsVersion":"15916"}}
{"level":"info","ts":1567081072.8198612,"logger":"baremetalhost_ironic","msg":"validating management access","host":"openshift-master-2"}
{"level":"info","ts":1567081072.8438115,"logger":"baremetalhost_ironic","msg":"found existing node by ID","host":"openshift-master-2"}

$ oc get baremetalhosts -n openshift-machine-api   openshift-master-0 -o yaml
apiVersion: metal3.io/v1alpha1
kind: BareMetalHost                         
metadata: 
  annotations:    
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"metal3.io/v1alpha1","kind":"BareMetalHost","metadata":{"annotations":{},"name":"openshift-master-0","namespace":"openshift-machine-api"},"spec":{"bmc":{"address":"ipmi://r730xd-c13u23-drac.example.com
","credentialsName":"openshift-master-0-bmc-secret"},"bootMACAddress":"xx","consumerRef":{"apiVersion":"machine.openshift.io/v1beta1","kind":"Machine","name":"kni1-master-0","namespace":"openshift-machine-api"},"externallyP
rovisioned":true,"online":true}}
  creationTimestamp: "2019-08-29T11:46:58Z"
  finalizers:                         
  - baremetalhost.metal3.io    
  generation: 2                                   
  name: openshift-master-0
  namespace: openshift-machine-api
  resourceVersion: "20447"
  selfLink: /apis/metal3.io/v1alpha1/namespaces/openshift-machine-api/baremetalhosts/openshift-master-0
  uid: b5432b0b-ca52-11e9-9446-525400264466
spec:
  bmc:
    address: ipmi://r730xd-c13u23-drac.example.com
    credentialsName: openshift-master-0-bmc-secret
  bootMACAddress: xx
  consumerRef:
    apiVersion: machine.openshift.io/v1beta1
    kind: Machine
    name: kni1-master-0
    namespace: openshift-machine-api
  externallyProvisioned: true
  hardwareProfile: default
  image:
    checksum: http://172.22.0.3:6180/images/rhcos-42.80.20190827.1-openstack.qcow2/rhcos-42.80.20190827.1-compressed.qcow2.md5sum
    url: http://172.22.0.3:6180/images/rhcos-42.80.20190827.1-openstack.qcow2/rhcos-42.80.20190827.1-compressed.qcow2
  online: true
  userData:
    name: master-user-data
    namespace: openshift-machine-api
status:
  errorMessage: ""
  goodCredentials:
    credentials:
      name: openshift-master-0-bmc-secret
      namespace: openshift-machine-api
    credentialsVersion: "9534"
  hardwareProfile: ""
  lastUpdated: "2019-08-29T12:17:52Z"
  operationalStatus: OK
  poweredOn: false
  provisioning:
    ID: a6da46f0-4c4f-437a-9ccb-0157ecf59899
    image:
      checksum: ""
      url: ""
    state: registering
  triedCredentials:
    credentials:
      name: openshift-master-0-bmc-secret
      namespace: openshift-machine-api
    credentialsVersion: "15911"

[hardys]

What state are the hosts in ironic?

Try this on the provisioning host (where openstackclient will need to be installed if not already done via dev-scripts):

export OS_AUTH_TYPE=none
export OS_ENDPOINT=http://172.22.0.3:6385/
openstack baremetal node list

[e-minguez]

$ openstack baremetal node list
+--------------------------------------+--------------------+---------------+-------------+--------------------+-------------+
| UUID                                 | Name               | Instance UUID | Power State | Provisioning State | Maintenance |
+--------------------------------------+--------------------+---------------+-------------+--------------------+-------------+
| a6da46f0-4c4f-437a-9ccb-0157ecf59899 | openshift-master-0 | None          | power on    | adopt failed       | False       |
| 5b01e590-8894-42d4-88c6-bf7bd5189496 | openshift-master-1 | None          | power on    | adopt failed       | False       |
| 93652777-ffd4-422c-bcca-a4c0dabc8612 | openshift-master-2 | None          | power on    | adopt failed       | False       |
+--------------------------------------+--------------------+---------------+-------------+--------------------+-------------+

[hardys]

We live-debugged this and it seems the RHCOS version was outdated wrt the installer, #778 will help with that, but I think if this works with the latest dev-scripts (including #766) then this can be closed.

[e-minguez]

I can confirm it works now using c5c69f8