libvirt: Network doesn't do dhcp

[cgwalters]

I'd like to be able to reboot my desktop overall, as well as just my cluster VMs.

It seems like the libvirt network that gets installed doesn't have DHCP set up, at least sometimes; is Terraform assigning static IPs? My VMs get stuck in the initramfs waiting for DHCP.

[crawford]

The installer uses libvirt to configure static DHCP leases for each machine. The network should be configured to auto-start, but maybe something on your system prevents that. Can you check that the dnsmasq instance for the cluster is running?

[cgwalters]

dnsmasq is running. When I investigated this a bit...I think what might be happening is the hostname is set via Ignition and dhclient uses that in the initial request, but the hostname doesn't persist into the initramfs, so on the next reboot we won't set it in the dhcp request or so?

Does rebooting work for you?

[crawford]

I'm able to reboot both of the nodes in the cluster as well as my desktop. Is libvirt configured to shutdown or hibernate VMs when your system goes down? I'm guessing, like my machine, yours also shuts down the VMs.

I'm on NixOS 18.09.1381.c1427bf45ff, for what it's worth.

    libvirtd = {
      enable = true;
      onShutdown = "shutdown";
      extraConfig = ''
        listen_tls = 0
        listen_tcp = 1
        auth_tcp = "none"
      '';
      extraOptions = [ "--listen" ];
    };

[jihed]

Hi,
I am having the same issue and am running on the same configuration as you list @crawford but on CentOS Linux release 7.5.1804 (Core).

Still investigating.

[cgwalters]

Hmm, it works for me when I destroyed my cluster and reinstalled. Offhand...I think this might have been something leftover from my tt0 network? The installer changed relatively recently to name the libvirt network the same as the cluster right?

Could be there's some bug/race condition in Terraform? I don't know Terraform well enough to debug.

For reference my current network (which has a <dhcp> element):

# virsh net-dumpxml osiris
<network>
  <name>osiris</name>
  <uuid>00cc746f-6664-449b-bd37-7f07011b9587</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='tt0' stp='on' delay='0'/>
  <mac address='52:54:00:ea:3c:45'/>
  <domain name='verbum.local' localOnly='yes'/>
  <dns>
    <srv service='etcd-server-ssl' protocol='tcp' domain='osiris.verbum.local' target='osiris-etcd-0.verbum.local' port='2380' weight='10'/>
    <host ip='192.168.126.10'>
      <hostname>osiris-api</hostname>
    </host>
    <host ip='192.168.126.11'>
      <hostname>osiris-api</hostname>
      <hostname>osiris-etcd-0</hostname>
    </host>
    <host ip='192.168.126.50'>
      <hostname>osiris</hostname>
    </host>
  </dns>
  <ip family='ipv4' address='192.168.126.1' prefix='24'>
    <dhcp>
      <host mac='4a:ea:77:33:ab:a0' name='osiris-master-0' ip='192.168.126.11'/>
      <host mac='e6:9b:19:f1:89:02' name='osiris-bootstrap' ip='192.168.126.10'/>
      <host mac='0a:33:4c:6e:cc:93' name='osiris-worker-0-jsrxw' ip='192.168.126.51'/>
    </dhcp>
  </ip>
</network>

@jihed can you paste your network?

[jihed]

Hi, I tried to fix the ip@ given by the installer on the network description. But, when the machines reboot, they got a different ip.
Here's my modification:

<network connections='2'>
  <name>ocp4</name>
  <uuid>279ce454-0e37-4834-b149-e503fc470e49</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='tt0' stp='on' delay='0'/>
  <mac address='52:54:00:c4:9c:5f'/>
  <domain name='openshift.testing' localOnly='yes'/>
  <dns>
    <srv service='etcd-server-ssl' protocol='tcp' domain='ocp4.openshift.testing' target='ocp4-etcd-0.openshift.testing' port='2380' weight='10'/>
    <host ip='192.168.126.11'>
      <hostname>ocp4-api</hostname>
      <hostname>ocp4-etcd-0</hostname>
    </host>
    <host ip='192.168.126.50'>
      <hostname>ocp4</hostname>
    </host>
  </dns>
  <ip family='ipv4' address='192.168.126.1' prefix='24'>
    <dhcp>
      <range start='192.168.126.10' end='192.168.126.100'/>
      <host mac='d6:50:85:79:34:d2' name='ocp4-master-0' ip='192.128.126.11'/>
      <host mac='46:d3:1f:a0:ce:1b' name='ocp4-worker-0-jz9bc' ip='192.128.126.50'/>
    </dhcp>
  </ip>

For example: I got 192.128.126.60 for the master.

Regards,

[wking]

The installer changed relatively recently to name the libvirt network the same as the cluster right?

This happened in ec68d77 (#660).

[dustymabe]

I'm hitting this too. After a shutdown and reboot of my host and virsh start of my cluster nodes they aren't getting DHCP.

[cgwalters]

I hit this even without suspending/rebooting my host. Just rebooting one of my workers caused this somehow. Or maybe...it's some sort of race condition and I was missing the <dhcp> entry after I did the install?

[cgwalters]

OK yeah, this doesn't actually seem related to suspend/shutdown, I just did two installs in a row, and looking at their network config I see:

  <ip family='ipv4' address='192.168.126.1' prefix='24'>
  </ip>

This must be something more like a bug in Terraform or the libvirt stack?

[steven-ellis]

I'm seeing similar issues trying to run this on a RHEL8 system. I can see the DNS requests via tcpdump, but they aren't being serviced correctly

 tcpdump   -i tt0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tt0, link-type EN10MB (Ethernet), capture size 262144 bytes
19:08:37.277822 IP6 :: > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
19:08:38.264249 IP6 :: > ff02::1:ff3f:5f0f: ICMP6, neighbor solicitation, who has fe80::644f:16ff:fe3f:5f0f, length 32
19:08:38.264258 IP6 :: > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
19:08:39.260253 IP6 fe80::644f:16ff:fe3f:5f0f > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
19:08:39.260263 IP6 fe80::644f:16ff:fe3f:5f0f > ff02::2: ICMP6, router solicitation, length 16
19:08:39.290899 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 66:4f:16:3f:5f:0f (oui Unknown), length 300
19:08:40.209908 IP6 fe80::644f:16ff:fe3f:5f0f > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
19:08:43.267971 IP6 fe80::644f:16ff:fe3f:5f0f > ff02::2: ICMP6, router solicitation, length 16
19:08:43.884266 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 66:4f:16:3f:5f:0f (oui Unknown), length 300
19:08:47.276273 IP6 fe80::644f:16ff:fe3f:5f0f > ff02::2: ICMP6, router solicitation, length 16
19:08:54.508247 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 66:4f:16:3f:5f:0f (oui Unknown), length 300
19:09:12.403258 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 66:4f:16:3f:5f:0f (oui Unknown), length 300

If I switch the VM to the default network I can see the DNS request being serviced correctly

# tcpdump   -i virbr0
19:11:43.965362 IP6 :: > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
19:11:43.979250 IP6 :: > ff02::1:ff3f:5f0f: ICMP6, neighbor solicitation, who has fe80::644f:16ff:fe3f:5f0f, length 32
19:11:44.615442 IP6 :: > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
19:11:44.979390 IP6 fe80::644f:16ff:fe3f:5f0f > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
19:11:44.979395 IP6 fe80::644f:16ff:fe3f:5f0f > ff02::2: ICMP6, router solicitation, length 16
19:11:45.881327 IP6 fe80::644f:16ff:fe3f:5f0f > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
19:11:46.297758 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 66:4f:16:3f:5f:0f (oui Unknown), length 300
19:11:46.300184 ARP, Request who-has 192.168.122.58 tell rhel8.mytest.io, length 28
19:11:47.359243 ARP, Request who-has 192.168.122.58 tell rhel8.mytest.io, length 28
19:11:48.383245 ARP, Request who-has 192.168.122.58 tell rhel8.mytest.io, length 28
19:11:48.983442 IP6 fe80::644f:16ff:fe3f:5f0f > ff02::2: ICMP6, router solicitation, length 16
19:11:49.322094 IP rhel8.mytest.io > 192.168.122.58: ICMP echo request, id 26745, seq 0, length 28
19:11:49.322143 IP rhel8.mytest.io.bootps > 192.168.122.58.bootpc: BOOTP/DHCP, Reply, length 300
19:11:49.323365 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 66:4f:16:3f:5f:0f (oui Unknown), length 300
19:11:49.323870 IP rhel8.mytest.io.bootps > 192.168.122.58.bootpc: BOOTP/DHCP, Reply, length 300
19:11:49.412480 ARP, Request who-has 192.168.122.58 (Broadcast) tell 0.0.0.0, length 28
19:11:50.411474 ARP, Request who-has 192.168.122.58 (Broadcast) tell 0.0.0.0, length 28
19:11:52.991368 IP6 fe80::644f:16ff:fe3f:5f0f > ff02::2: ICMP6, router solicitation, length 16

[steven-ellis]

I've traced the issue to the bridge entry in the network definition

  <bridge name='tt0' stp='on' delay='0'/>

If I use ttX for the bridge then DHCP doesn't work correctly. If I redefine the network with a virbrX entry then DHCP work fine under a RHEL8 host

  <bridge name='virbr1' stp='on' delay='0'/>