[galternat]

Problem

With systemd-logind, an input device hotplugged while switching back to the
server's VT can leave X in one of these states:

• it never re-enters the VT: the other VT's contents stay on screen, input is
  dead, and further VT switches hang, since the server still owns the VT;
• it re-enters, but the input devices that were active before the switch stay
  disabled (only the freshly hotplugged one works);
• or nothing goes wrong, depending on timing.

The log has systemd-logind: got pause for ... lines but some or all of the
matching got resume for ... lines are missing. I hit this in a VM where some
devices are added automatically by the VM agent during VT switches, but it
reproduces on real systems too, with much worse consequences since this
can leave the machine permanently stuck with no input working.

Cause

socket_handler() only runs when the system bus socket is readable, and
dbus_connection_read_write_dispatch() either dispatches a queued message or
reads, never both. Every ResumeDevice carries an FD, so the kernel returns
one per recvmsg(), and the handler gets through one signal per main loop
pass. When config/udev handles the hotplug later in the same pass, its
TakeDevice call (dbus_connection_send_with_reply_and_block()) reads the
remaining signals into libdbus' queue while waiting for its reply. The socket
is then empty, and nothing dispatches them until some unrelated message
arrives.

Confirmed by bpftrace on an unpatched server, switching back to its VT:

29.451 recvmsg(bus) 196 bytes from socket_handler    <- one ResumeDevice (226:1)
29.452 systemd_logind_vtenter()
29.466 udev socket_handler                           <- same pass: mouse "add"
29.468 >> blocking call TakeDevice
29.469 recvmsg(bus) 196 bytes INSIDE blocking call   <- x6: the input ResumeDevices
29.471 recvmsg(bus) 80 bytes INSIDE blocking call    <- the TakeDevice reply

After that the bus socket is never reported readable again.

Changes

1. config: dbus-core: dispatch messages queued during blocking calls
   A dispatch status function queues a work proc whenever libdbus' queue
   becomes non-empty, including inside another caller's blocking call. Work
   procs run at the top of every WaitForSomething() iteration, before the
   server can block again; that also holds after timer callbacks such as
   reconnect_timer(), since check_timers() returns 0 when it ran a timer.
2. config: dbus-core: poll our own duplicate of the bus FD
   libdbus closes its socket as soon as it sees the bus go away, before
   Disconnected is queued. The closed socket drops out of epoll, so
   Disconnected was never dispatched and the server never reconnected. The
   stale registration also makes a client accepted on the recycled number hang
   (ospoll_add() only updates the existing entry), and once teardown does
   run, which commit 1 makes possible, RemoveNotifyFd() would free that
   client's data as a struct notify_fd. Polling a duplicate owned by
   dbus-core keeps the number reserved until teardown and keeps the hangup
   visible to socket_handler().
3. config: dbus-core: don't leak the connection on disconnect
   message_filter() cleared bus_info.connection before calling
   teardown(), so the reference from dbus_bus_get() was never dropped and
   every bus restart leaked the old connection. This predates the series, but
   until commit 2 the Disconnected path hardly ever ran.
4. xfree86: systemd-logind: ignore messages without a path
   A reply arriving after its blocking call timed out goes to the filter, and
   dbus_message_get_path() returns NULL into strcmp().
5. xfree86: systemd-logind: send PauseDeviceComplete for the paused device
   message_filter() passed major/minor in the opposite order to
   systemd_logind_ack_pause(), so the acknowledgement named the wrong device.
   logind only asks for a graceful pause on seats without VTs, where the
   session switch then never completes.

Reproducer

With a logind-managed X on VT1 and a text console on VT2, run the following
from VT2 as root:

chvt 1; ./uinput-hold 10 1

This creates new input devices right after switching to VT1 and should trigger
the bug. Make sure you have no unsaved work, as a reboot may be required after
this. I managed to recover a few times by running sysrq+K on VT1 and switching
to a live X server on another VT (switching to VT2 didn't work), but I wouldn't rely
on it.

Here's the required uinput-hold.c:

// Creates argv[2] uinput mice (default 1), holds them for argv[1] seconds (default 5).
#include <err.h>
#include <fcntl.h>
#include <linux/uinput.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/ioctl.h>
#include <unistd.h>

int main(int argc, char **argv)
{
    int secs = argc > 1 ? atoi(argv[1]) : 5;
    int count = argc > 2 ? atoi(argv[2]) : 1;

    for (int i = 0; i < count; i++) {
        struct uinput_setup us = {
            .id = { .bustype = BUS_VIRTUAL, .vendor = 0x1234, .product = 0x5678 },
        };
        int fd = open("/dev/uinput", O_WRONLY | O_NONBLOCK);

        if (fd < 0)
            err(1, "/dev/uinput");
        ioctl(fd, UI_SET_EVBIT, EV_KEY);
        ioctl(fd, UI_SET_KEYBIT, BTN_LEFT);
        ioctl(fd, UI_SET_EVBIT, EV_REL);
        ioctl(fd, UI_SET_RELBIT, REL_X);
        ioctl(fd, UI_SET_RELBIT, REL_Y);
        snprintf(us.name, sizeof(us.name), "vtstrand mouse %d-%d", getpid(), i);
        if (ioctl(fd, UI_DEV_SETUP, &us) < 0 || ioctl(fd, UI_DEV_CREATE) < 0)
            err(1, "uinput create");
    }
    sleep(secs);
}

Compile via gcc -std=c99 -Wall -o uinput-hold uinput-hold.c.

Tests

The current master (483a172) X hangs quite reliably on my setup. With the
patch applied, it finishes the switch correctly even after trying it multiple times.
Clean shutdown (SIGTERM) still runs teardown and closes the duplicate. Commits
4 and 5 were checked by inspection, as they are trivial related bug fixes.

[galternat]

Apparently, this was hit by other users during suspend/resume, see https://gitlab.freedesktop.org/xorg/xserver/-/issues/1915, this should fix that too.