Sled-agent crashed after hitting `ipadm: set-ifprop: mtu: Interface does not exist`

[askfongjojo]

Here is the relevant portion of the sled-agent log file:

21:38:19.879Z INFO SledAgent (InstanceManager): Publishing instance state update to Nexus
    instance_id = c59ec866-fb95-4267-b976-9de840a73157
    state = InstanceRuntimeState { run_state: Failed, sled_id: 80395ead-2a26-4718-9919-cda27cad128f, propolis_id: 7931d3fb-5a0f-4e8a-83ed-554b72a05750, dst
_propolis_id: None, propolis_addr: Some([fd00:1122:3344:102::22]:12400), migration_id: None, propolis_gen: Generation(1), ncpus: InstanceCpuCount(8), memor
y: ByteCount(17179869184), hostname: "vm-8c-16m", gen: Generation(3), time_updated: 2023-07-06T21:38:19.879282812Z }
21:38:19.935Z INFO SledAgent (dropshot (SledAgent)): request completed
    error_message_external = Internal Server Error
    error_message_internal = Zone booted, but running a command experienced an error: Error running command in zone 'oxz_propolis-server_7931d3fb-5a0f-4e8a
-83ed-554b72a05750': Command [/usr/sbin/ipadm set-ifprop -t -p mtu=9000 -m ipv6 oxControlInstance0] executed and failed with status: exit status: 1  stdout
: \n  stderr: ipadm: set-ifprop: mtu: Interface does not exist
    local_addr = [fd00:1122:3344:102::1]:12345
    method = PUT
    remote_addr = [fd00:1122:3344:102::3]:33463
    req_id = fa4defff-9059-4b36-ab2d-a0746a7d92db
    response_code = 500
    uri = /instances/c59ec866-fb95-4267-b976-9de840a73157/state
21:38:20.030Z INFO SledAgent (dropshot (SledAgent)): accepted connection
    local_addr = [fd00:1122:3344:102::1]:12345
    remote_addr = [fd00:1122:3344:102::3]:47155
21:38:20.030Z WARN SledAgent (InstanceManager): Halting and removing zone: oxz_propolis-server_7931d3fb-5a0f-4e8a-83ed-554b72a05750
    instance_id = c59ec866-fb95-4267-b976-9de840a73157
thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: AdmError { op: Uninstall, zone: "oxz_propolis-server_7931d3fb-5a0f-
4e8a-83ed-554b72a05750", err: CommandOutput(CommandOutputError("exit code 1\nstdout:\n\nstderr:\nzoneadm: zone 'oxz_propolis-server_7931d3fb-5a0f-4e8a-83ed
-554b72a05750': uninstall operation is invalid for shutting_down zones.")) }', sled-agent/src/instance.rs:533:64
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

The complete log file can be found at: catacomb.eng.oxide.computer:/data/staff/dogfood/jul-6/sled-agent-gc17.log

[jmpesp]

This is due to the changes I made to set MTUs, and their conflict with self-assembling zones. Propolis recently change to be self-assembling, and includes

ipadm delete-if "$DATALINK" || true

in the method script. This races with

omicron/illumos-utils/src/running_zone.rs

Lines 395 to 443 in c3e856e

	if !site_profile_xml_exists {
	let args = vec![
	IPADM.to_string(),
	"create-if".to_string(),
	"-t".to_string(),
	vnic.clone(),
	];
	running_zone.run_cmd(args)?;
	} else {
	// If the zone is self-assembling, then it's possible that the IP
	// interface does not exist yet because it has not been brought up
	// by the software in the zone. Run `create-if` here, but eat the
	// error if there is one: this is safe unless the software that's
	// part of self-assembly inside the zone is also trying to run
	// `create-if` (instead of `create-addr`), and required for the
	// `set-ifprop` commands below to pass.
	let args = vec![
	IPADM.to_string(),
	"create-if".to_string(),
	"-t".to_string(),
	vnic.clone(),
	];
	let _result = running_zone.run_cmd(args);
	}
	let commands = vec![
	vec![
	IPADM.to_string(),
	"set-ifprop".to_string(),
	"-t".to_string(),
	"-p".to_string(),
	format!("mtu={}", CONTROL_VNIC_MTU),
	"-m".to_string(),
	"ipv4".to_string(),
	vnic.clone(),
	],
	vec![
	IPADM.to_string(),
	"set-ifprop".to_string(),
	"-t".to_string(),
	"-p".to_string(),
	format!("mtu={}", CONTROL_VNIC_MTU),
	"-m".to_string(),
	"ipv6".to_string(),
	vnic,
	],
	];

, leading to this bug.

[jmpesp]

Opened oxidecomputer/crucible#828 and oxidecomputer/propolis#458 to fix the manifest scripts

[smklein]

#3454 tracks preventing the sled agent crashing during instance provisioning, regardless of error source - I can work on this next.