Gateway k3s-agent crash-loops on flannel vxlan reCreateVxlan panic (index out of range)

[pau-hedgehog]

During env-1 mesh/l2vni release-test run 20260728-113711-release-test-default-default-ce28f3b58-pau (build ema/test-connectivity @ ce28f3b58, k3s v1.36.2+k3s1, flannel v0.28.4, gateway dataplane v0.22.0), both gateway VMs hit a repeating k3s-agent crash on the flannel vxlan backend.

Verbatim from gateway-2's journal:

Jul 28 12:17:26 gateway-2 k3s[11290]: I0728 12:17:26.083720   11290 vxlan_network.go:143] Interface flannel.1 deleted
Jul 28 12:17:26 gateway-2 k3s[11290]: I0728 12:17:26.083791   11290 vxlan_network.go:102] vxlan device missing, attempting to recreate...
Jul 28 12:17:26 gateway-2 k3s[11290]: panic: runtime error: index out of range [0] with length 0
Jul 28 12:17:26 gateway-2 k3s[11290]: goroutine 9177 [running]:
Jul 28 12:17:26 gateway-2 k3s[11290]: github.com/flannel-io/flannel/pkg/backend/vxlan.(*network).reCreateVxlan(0x196b67d3400, {0x8522398, 0x196b5868d80})
Jul 28 12:17:26 gateway-2 k3s[11290]:         /go/pkg/mod/github.com/flannel-io/flannel@v0.28.4/pkg/backend/vxlan/vxlan_network.go:217 +0xda6
Jul 28 12:17:26 gateway-2 k3s[11290]: github.com/flannel-io/flannel/pkg/backend/vxlan.(*network).Run.func3()
Jul 28 12:17:26 gateway-2 k3s[11290]:         /go/pkg/mod/github.com/flannel-io/flannel@v0.28.4/pkg/backend/vxlan/vxlan_network.go:106 +0x25
Jul 28 12:17:26 gateway-2 k3s[11290]: created by github.com/flannel-io/flannel/pkg/backend/vxlan.(*network).Run in goroutine 740
Jul 28 12:17:26 gateway-2 k3s[11290]:         /go/pkg/mod/github.com/flannel-io/flannel@v0.28.4/pkg/backend/vxlan/vxlan_network.go:105 +0x3a5
Jul 28 12:17:26 gateway-2 systemd[1]: k3s-agent.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Jul 28 12:17:26 gateway-2 systemd[1]: k3s-agent.service: Failed with result 'exit-code'.

systemd auto-restarts the agent after each crash. This recurred 28 times on gateway-2 and 10 times on gateway-1 over roughly 50 minutes in this one run. Each crash also drops the agent's tunnel session to control-1 for a few seconds, visible as bursts of failed to find Session for client gateway-<N> 502s in the control-1 k3s log at matching timestamps. Workload containers (dataplane, frr) are unaffected, only the k3s-agent/kubelet process bounces.

Confirmed: flannel's reCreateVxlan indexes into an empty slice when reprogramming flannel.1 after observing it deleted, panicking.

Not confirmed: what repeatedly deletes the flannel.1 interface on the gateway node in the first place.

Ask: (1) guard against the empty-slice case in the vendored flannel reCreateVxlan path so it doesn't panic, (2) investigate what deletes the gateway's flannel.1 vxlan interface repeatedly.

Where in the attached show-tech bundle: multi-vpc-single-subnet-suite/gateway-peering-masquerade-and-port-forward-nat/gateway-2-show-tech.log (28 occurrences) and gateway-1-show-tech.log (10 occurrences), grep for panic: runtime error: index out of range. The matching tunnel-session 502s are in control-1-show-tech.log, grep for failed to find Session for client gateway-.

show-tech.tar.gz

[Frostman]

Seems relevant:

• Index out of range panic in flannel vxlan recreate for IPv4-only setups flannel-io/flannel#2433 which points to Fix: recreate VXLAN device (flannel.*) when external interface is deleted and re-added flannel-io/flannel#2272 that introduced the behavior but we're on that version for a long time
• fix(vxlan): guard ifaceAddrsV6[0] on IPv4-only setups flannel-io/flannel#2434 supposeduly fixes it, but it's recent and k3s didn't update to that flannel

IMO likely it's caused by dataplane messing with the vxlan devices as systemd-networkd config is static for mgmt NIC

[pau-hedgehog]

Confirmed this is still present on master (dataplane v0.23.1, k3s v1.36.2+k3s1), not specific to the ema/test-connectivity build (dataplane v0.22.0) it was first seen on.

Live-watched both gateways' journalctl -u k3s-agent -f during env-ctl run 20260728-155251-release-test-default-default-master-pau (hhfab master @ fc58a6d48). The panic recurred twice within the first ~20 minutes after the gateway VMs came up, both times on gateway-1 and gateway-2 within seconds of each other:

Jul 28 15:59:35 gateway-1 k3s[5893]: I0728 15:59:35.847673    5893 vxlan_network.go:143] Interface flannel.1 deleted
Jul 28 15:59:35 gateway-1 k3s[5893]: panic: runtime error: index out of range [0] with length 0
Jul 28 15:59:35 gateway-1 systemd[1]: k3s-agent.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Jul 28 15:59:40 gateway-1 systemd[1]: k3s-agent.service: Scheduled restart job, restart counter is at 1.

Jul 28 15:59:39 gateway-2 k3s[5956]: I0728 15:59:39.082922    5956 vxlan_network.go:143] Interface flannel.1 deleted
Jul 28 15:59:39 gateway-2 k3s[5956]: panic: runtime error: index out of range [0] with length 0
Jul 28 15:59:39 gateway-2 systemd[1]: k3s-agent.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Jul 28 15:59:45 gateway-2 systemd[1]: k3s-agent.service: Scheduled restart job, restart counter is at 1.

Jul 28 16:15:16 gateway-1 k3s[8327]: I0728 16:15:16.687984    8327 vxlan_network.go:143] Interface flannel.1 deleted
Jul 28 16:15:16 gateway-1 k3s[8327]: panic: runtime error: index out of range [0] with length 0
Jul 28 16:15:16 gateway-1 systemd[1]: k3s-agent.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Jul 28 16:15:21 gateway-1 systemd[1]: k3s-agent.service: Scheduled restart job, restart counter is at 2.

Jul 28 16:15:16 gateway-2 k3s[8670]: I0728 16:15:16.708492    8670 vxlan_network.go:143] Interface flannel.1 deleted
Jul 28 16:15:16 gateway-2 k3s[8670]: panic: runtime error: index out of range [0] with length 0
Jul 28 16:15:16 gateway-2 systemd[1]: k3s-agent.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Jul 28 16:15:21 gateway-2 systemd[1]: k3s-agent.service: Scheduled restart job, restart counter is at 2.

Both gateways hitting it within seconds of each other, twice, points away from a per-VM hardware/timing fluke and toward something environmental or config-driven happening to both gateway VMs at once. Still don't know what deletes flannel.1 in the first place.

[pau-hedgehog]

The panic itself has a known upstream cause, flannel-io/flannel#2434 (merged 2026-04-27, fixed in v0.28.5+, still open in the vendored v0.28.4 here): reCreateVxlan() indexes ifaceAddrsV6[0] unconditionally even when IPv6 is disabled and the slice is empty, so any recreate attempt panics. That explains why this crashes instead of failing gracefully, but not why flannel.1 disappears in the first place.

That part looks like it's on the dataplane side. Confirmed via flannel.1's netlink watcher, which fires strictly on update.Attrs().Name == "flannel.1" with update.Header.Type == unix.RTM_DELLINK, so this is a genuine kernel-level deletion, not flannel misreading an unrelated event. Something is actually removing flannel.1, and the timing across the occurrences we've seen lines up with the gateway dataplane container's own interface setup/restart activity on the same node. Could someone from dataplane look at whether the interface manager (or its init/setup path) is touching or resetting flannel.1, e.g. during its own device enumeration or TAP/vxlan interface handling?

[Frostman]

yeah, and if it's noone removing flannel.1 we'll not even end up in the code path for this panic. so the fix is to fix dataplane to not remove the flannel.1 and hopefully k3s would bump the flannel soon

[pau-hedgehog]

Tested PR #1671 (fix(interface-manager): Do not delete interfaces we did not create) on env-1 mesh/l2vni release-test.

$ ./env-ctl.sh status
=== run 20260729-081833-release-test-v0-9ee6ed90e-release-default-master-pau ===
hhfab:     master @ 2581aa3b3
dataplane: pr/daniel-noland/fix-interface-manager @ 9ee6ed90e
wiring:    master @ f9dedea
iperfs-speed: env-1=2000

env         topology       mode     phase           result                                runtime  last-event
env-1       mesh           l2vni    report          release-test - PASS 39/46 (7 skip)    1h36m    09:56 UTC run end rc=0

Collected show-tech right after (hhfab-master vlab show-tech) and checked gateway-1 and gateway-2 for the same three signals as the original report:

$ grep -c 'panic:' gateway-1-show-tech.log gateway-2-show-tech.log
gateway-1-show-tech.log:0
gateway-2-show-tech.log:0

$ grep -c 'vxlan device missing' gateway-1-show-tech.log gateway-2-show-tech.log
gateway-1-show-tech.log:0
gateway-2-show-tech.log:0

$ grep -c 'external interface.*not found, retrying' gateway-1-show-tech.log gateway-2-show-tech.log
gateway-1-show-tech.log:0
gateway-2-show-tech.log:0

The flannel retry-loop message never fires either, not just the panic, meaning flannel.1 was never deleted in this run.

k3s-agent.service status on both gateways confirms a single continuous process for the whole test window, no restarts:

$ grep -A9 '=== k3s-agent.service status ===' gateway-1-show-tech.log
=== k3s-agent.service status ===
● k3s-agent.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s-agent.service; enabled; preset: disabled)
     Active: active (running) since Wed 2026-07-29 08:23:35 UTC; 1h 36min ago
 Invocation: 427e48b3090d4debaf2ffa6a31a55854
       Docs: https://k3s.io
    Process: 6511 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
    Process: 6512 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
   Main PID: 6514 (k3s-agent)
      Tasks: 107

show-tech-gw.tar.gz

[mvachhar]

I believe resolving #1670 will fix this.