Instance creation failing during link creation (OPTE ioctl)

[smklein]

When following the remote-access preview instructions, I've noticed flakiness creating instances with the terraform workflow.

I've been following the demo instructions on folgers, and when I get to the point of creating instances via terraform, I use: terraform init && terraform apply. For most instances, this seems to work, but I occasionally see one or two which fail with a "500 internal error".

Digging into the sled agent logs, I see the following:

[2022-09-07T15:34:00.571064026Z]  INFO: SledAgent/dropshot (SledAgent)/12362 on folgers: accepted connection (local_addr=[fd00:1122:3344:101::1]:12345, remote_addr=[fd00:1122:3344:101::3]:57942)                                            
[2022-09-07T15:34:00.571380893Z]  INFO: SledAgent/InstanceManager/12362 on folgers: instance_ensure e7670ef9-73ec-4846-8058-f84a263e2ef9 -> InstanceRuntimeStateRequested { run_state: Running, migration_params: None }                      
[2022-09-07T15:34:00.571637528Z]  INFO: SledAgent/InstanceManager/12362 on folgers: new instance                                                                                                                                              
[2022-09-07T15:34:00.572034446Z]  INFO: SledAgent/InstanceManager/12362 on folgers: Instance::new w/initial HW: InstanceHardware { runtime: InstanceRuntimeState { run_state: Creating, sled_id: fb0f7546-4d46-40ca-9d56-cbb810684ca7, propoli
s_id: e685ef90-d155-4bcd-abf3-12531e6c1ef4, dst_propolis_id: None, propolis_addr: Some([fd00:1122:3344:101::e]:12400), migration_id: None, ncpus: InstanceCpuCount(4), memory: ByteCount(2147483648), hostname: "db-instance-1", ...
[2022-09-07T15:34:00.572289002Z]  INFO: SledAgent/dropshot (SledAgent)/12362 on folgers: request completed (req_id=4b7b08f2-c1b6-4574-810f-f540ee5715c9, uri=/instances/e7670ef9-73ec-4846-8058-f84a263e2ef9, method=PUT, remote_addr=[fd00:11
22:3344:101::3]:57942, local_addr=[fd00:1122:3344:101::1]:12345, error_message_external="Internal Server Error", response_code=500)
    error_message_internal: Error managing instances: Instance error: Failure interacting with the OPTE ioctl(2) interface: netadm failed dlmgmtd: link id creation failed: 17

I'm using Omicron @ 55cc15c

[smklein]

This issue seems flaky, but I can reproduce it. Ben, what diagnostics could I provide from my machine to help?

[leftwo]

Could this be related to: oxidecomputer/netadm-sys#8
I saw a similar thing on folgers last week. Some comments from @rzezeski in chat:

so, this isn't the easiest thing to understand because we have nesting of errors, and I don't think sled-agent is quite right to label this as part of the ioctl(2) interface, but I believe what happened here is the libnet::link::create_link_id() call inside of OpteHdl::create_xde() failed, causing the create_xde() method to exist early with no harm done. Then on the second try the link id creation was successful, and so create_xde() continued onto the actual ioctl call, and everything was successful.
As to why the create_link_id() call would fail, my guess is dlmgmtd may not have been up? But I'm not sure why it wouldn't be up.
oh, wait
the errno is 17 (EEXIST)`, so presumably that link name already existed
but something must have deleted it between calls
I think we could improve this error message in opte-ioctl by inspecting the libnet::Error and doing something like strerror for the errno
rpz
https://github.com/oxidecomputer/netadm-sys/blob/main/libnet/src/link.rs#L575
I think we could probably match against the response.err and maybe use the libnet::Error::AlreadyExists(String) value

[rzezeski]

The oxidecomputer/netadm-sys#8 is not the cause of the problem, it would just be a nice to have so that you get a human-friendly version of errno (instead of just 17). The problem is that, for whatever reason, omicron is trying to create links (OPTE Ports) that already exist. In another issue, #1682, there is discussion about these being links that are in some kind of half-created state, where there is a linkid entry in dlmgmtd, but not such link in OPTE/DLS. I think there are two questions:

1. How do we get into this state?

2. What fix/workaround should we implement based on (1). It seems [sled agent] Compare dladm results with datalink mgmt cache file #1685 is a workaround already in the works, but I don't think I've seen an answer to question (1).

[bnaecker]

Thanks @rzezeski. I don't believe we understand the cause of this situation. Somehow, we've either partially created or partially destroyed a link with that name. @rcgoodfellow and I were looking at the logic in xde for cleaning up resources. It looks like that should work, but it's possible that there's a bug there that keeps the defunct links around.