mvp vm attestation support in propolis-server (rfd 605)

[jordanhendricks]

This issue will cover #1010, which is a ticket for external tracking on the RFD 605 board. At the risk of being redundant, I'm creating a fresh issue to outline our design of an mvp attestation path in propolis-server and track incremental progress/PRs.

MVP Design Overview

We need a starting implementation of VM attestation as per RFD 605. (This is the relevant part of the RFD for us.)

The basic idea here is:

• a guest submits a 32-byte piece of data for attestation over a known attestation port
• this port is presented to the guest via a vsock (Want virtio-vsock support #1031) device
• on the propolis side of the port, propolis runs a "socket server" thread that can respond to attestation requests
• as part of the attestation process, we need additional qualifying_data tying the instance to this particular request. For our mvp targeting R19, that is only the hash of the instance boot disk, and its instance UUID.
• on instance start, propolis will hash the boot disk to be used in the qualifying data. This has two implications for our design: (1) that the boot disk is read-only ([nexus] create read-only disks from images or snapshots omicron#9731) and (2) that the boot disk must be hashed before we can start responding to attestation requests. As such, the socket server will fast-fail requests until the boot disk is hashed, so as to neither make instance start times obscenely long nor respond to attestation requests until we have the boot digest ready.
• propolis-server attests the data provided by the guest, along with the qualifying data, via the interfaces in oxidecomputer/vm-attest-proto
  
• for the MVP, communication to the RoT is via sled-agent APIs over the underlay network (sled-agent: add RoT attestation endpoints omicron#9739). This work is already plumbed through to the vm-attest-proto implementation such that propolis-server can use it (https://github.com/oxidecomputer/dice-util/blob/main/verifier/src/sled_agent.rs).

Incremental Work

We've largely been incrementally developing this feature in propolis-standalone, which allowed us to prove out the socket server and upstack consumers (i.e., software in a guest instance) without needing a whole control plane or even just sled-agent. This has been extremely useful for development and iterating quickly. For the real feature, we need support in propolis-server as well.

First we'll get propolis-standalone support merged, which will allow continued development and testing with propolis-standalone and allow us to start moving common code for -standalone and -server into lib/.

We have the following standalone PRs in progress from work so far:

• allow standalone to run a basic attestation server, using mock or real RoT (standalone: want ability to run attestation server backed by real or mock RoT #1068)
• hash the boot disk (standalone: want option to hash contents of boot disk #1071) and hook this up to the attestation server

For the propolis-server side, we need to do the following (these items may be broken up or combined into PRs depending on how it makes sense to land things):

• commonize socket server code for use by both server and standalone in lib/
• adapt boot disk hashing code for common use
• fetch instance ID out of instance spec
• plug in sled-agent dice_verifier::Attest impl to propolis-server: verifier: add sled-agent implementation for Attest dice-util#347
• Wire up vsock device to propolis-server #1069
• guest VMs should get vsock devices by default omicron#10000

[jordanhendricks]

Today we (@papertigers and @iximeow and I) achieved another important incremental development in this feature: we got a prototype version of propolis-server with an attestation server running on a racklette, and we saw an attestation work end-to-end from inside an instance all the way through to its sled's RoT (see below for caveats on what we did and didn't test):

root@test-instance:~# appraiser --root-cert ./jordan-test/test-root.cert.pem --reference-measurements ./jordan-test/corim.cbor.better --vm-instance-cfg vm-instance-cfg.json vm-instance-rot vsock 605
[2026-03-21T05:33:41Z INFO  appraiser] cert chain verified against root with CN: Platform Identity Staging Root A
[2026-03-21T05:33:41Z INFO  appraiser] attestation verified
[2026-03-21T05:33:41Z INFO  appraiser] measurement log from Oxide Platform RoT appraised
[2026-03-21T05:33:41Z INFO  appraiser] comparing reference vm instance cfg: VmInstanceConf { uuid: db5bf54c-48c5-4455-a1e1-6c7dfc26e351, boot_digest: Some(Sha256([190, 77, 244, 224, 133, 23, 95, 61, 224, 200, 172, 72, 55, 225, 194, 201, 163, 78, 137, 131, 32, 157, 172, 107, 84, 158, 148, 21, 79, 124, 221, 156])) }
[2026-03-21T05:33:41Z INFO  appraiser] to vm instance cfg from VmInstanceRot: VmInstanceConf { uuid: db5bf54c-48c5-4455-a1e1-6c7dfc26e351, boot_digest: Some(Sha256([190, 77, 244, 224, 133, 23, 95, 61, 224, 200, 172, 72, 55, 225, 194, 201, 163, 78, 137, 131, 32, 157, 172, 107, 84, 158, 148, 21, 79, 124, 221, 156])) }
[2026-03-21T05:33:41Z INFO  appraiser] metadata from Oxide VM Instance RoT appraised
[2026-03-21T05:33:41Z INFO  appraiser] appraised attestation from VmInstanceRot over vsock

The branch we deployed is: https://github.com/oxidecomputer/omicron/tree/jhendricks/rfd-605, which uses the propolis branch https://github.com/oxidecomputer/propolis/tree/jhendricks/rfd-605.

Technical milestones achieved with this branch

The POC we did today proved out several new technical milestones in this project:

• end-to-end communication between a guest and propolis-server via a vsock device with a listening thread on the host (attestation server)
• integration with sled-agent RoT communication (and the sled-agent attest trait implementation)
• proving out the vm-attest interfaces in the propolis-server architecture
• for the first time, we saw a successful attestation end to end, from a guest instance, through propolis-server, sled-agent, and the sled's RoT 🎉

Functionality to come / limitations

This branch hard-codes the boot-disk digest and instance ID that ships with the oxidecomputer/vm-attest-demo image. We also had to manually setup the root cert and reference measurements used in the image since the demo VM does not match what was running on our racklette.

For the root cert we did something like:

BRM42220004# verifier-cli --interface sled-agent --sled-addr '[fd69:aeb:bd87:101::1]:12345' cert-chain

Then pasted that into the VM, and manually edited the file to just include the root cert.

For the reference measurements, we nabbed the output from a failed attestation with the appraiser tool and generated a corim file by hand:

commands to generate logs

# output from `appraiser` inside the instance
[2026-03-21T05:25:41Z INFO  appraiser] attestation verified
Error: appraise platform attestation

Caused by:
    Measurements are not a subset of reference measurements: measurement set
     sha3-256;cdfde47ca96b935e46866e30be6add1bff2b9a904e08c2edd71eb56e2c78ce07
     sha3-256;23f984bf34dd3c98140dd3f49a1b1de9d620a2b52d73605a0db9e930ceb332b0

# KDL file to generate corim
$ cat madrid.kdl 
// This KDL describes a rats_corim::Corim instance from
// https://github.com/oxidecomputer/dice-util. Use the `attest-mock` tool to
// produce a CBOR encoding of it.
// ```shell
// $ attest-mock this-file.kdl corim
// ```
vendor "test"
tag-id "test"
id "test"

measurement {
    mkey "fwid-from-cert-chain"
    algorithm 10
    digest "cdfde47ca96b935e46866e30be6add1bff2b9a904e08c2edd71eb56e2c78ce07"
}

measurement {
    mkey "fwid-from-log"
    algorithm 10
    digest "23f984bf34dd3c98140dd3f49a1b1de9d620a2b52d73605a0db9e930ceb332b0"
}

# generate corim
jordan@rodman:~/src/dice-util$ ./target/debug/attest-mock madrid.kdl corim > corim.cbor

# massage into a file the guest can use
$ xxd -ps corim.cbor 
a200647465737401815892d901faa201a100647465737404a1008182a100
a101647465737482a20074667769642d66726f6d2d636572742d63686169
6e01a10281820a5820cdfde47ca96b935e46866e30be6add1bff2b9a904e
08c2edd71eb56e2c78ce07a2006d667769642d66726f6d2d6c6f6701a102
81820a582023f984bf34dd3c98140dd3f49a1b1de9d620a2b52d73605a0d
b9e930ceb332b0

# on the instance
root@test-instance:~/jordan-test# ^C
root@test-instance:~/jordan-test# echo "a200647465737401815892d901faa201a100647465737404a1008182a100a101647465737482a200647465737401815892d901faa201a100647465737404a1008182a100a101647465737482a20074667769642d66726f6d2d636572742d636861696e01a10281820a5820cdfde47ca96b935e46866e30be6add1bff2b9a904e08c2edd71eb56e2c78ce07a2006d667769642d66726f6d2d6c6f6701a10281820a582023f984bf34dd3c98140dd3f49a1b1de9d620a2b52d73605a0db9e930ceb332b0" | xxd -r -ps - > corim.cbor.better

# run the appraiser in the instance with the correct cert and measurements
$ appraiser --root-cert ./jordan-test/test-root.cert.pem --reference-measurements ./jordan-test/corim.cbor.better --vm-instance-cfg vm-instance-cfg.json vm-instance-rot vsock 605

There are several things we need to move from the experimental propolis-standalone branches into the propolis-server branch:

• thread performing the boot disk hash of the crucible disk (this design has been proved out, but it needs to be incorporated into the propolis-server architecture)
• use the actual instance ID
• figure out how to more seamlessly test with images that have the correct root cert and measurements for testing on racklettes
• some more cleanup of the prototype code

We ran into a number of issues along the way, some of which we've turned into PRs that are relevant for this project:

• impl Send for VmInstanceRoT safely vm-attest#57
• remove dependency on ipcc vm-attest#58
• verifier: humility isn't a tempfile dice-util#358

[flihp]

All 3 of the PRs linked above have been reviewed and since it's the weekend I'll do the merge as well to keep things moving.