OPTE integration

[smklein]

What follows is an overview of planned OPTE integration, after a conversation with @rzezeski. This also builds on top of functionality created by @bnaecker.

Initialization

• The Sled Agent is responsible for consuming requests from Nexus, and allocating instances. Currently, this includes allocation of VNICs using a particular representation of a NetworkInterface object.
  • In the long-term, this allocation should likely be replaced with an API to directly allocate VNICs from OPTE.
  • In the short-term, we'll continue allocating VNICs via the current mechanism (dladm create-vnic), but we'll use the ioctl-based interface from OPTE to perform operations like setting IP configuration and firewall rules.
• OPTE exposes a library interface for interacting with OPTE ports - the open_port invocation takes a VNIC name as input, and should return a connection to an OpteAdm object allowing for further configuration.
  • Note: The OpteAdm object is a bit overloaded; it may be split into more fine-grained objects as a result of OpteADM: Overloaded? opte#14
  • When Nexus communicates with the Sled agent to allocate an instance, it sends an object called InstanceHardware, representing the hardware which should be initialized for the instance. This includes a representation of the aforementioned NetworkInterface objects, which themselves contain "VPC ID", "Subnet ID", "Mac address", and "IP address". Although this representation is sufficient for the external API, as well as (probably?) storage in the database, it doesn't really work for shoving data into OPTE - at the point that this info is ready to be thrown at OPTE, it needs to be expanded (as an example: OPTE has no idea what the UUID of a VPC means. It needs to know all the information about that VPC which should be applied, since it won't be making upcalls to Nexus for further lookups).
  • Soooo once we get a more expanded InstanceHardware representation piped down to the Sled Agent, we can expand the VNIC representation to create an OpteAdm handle after the VNIC has been created, and make the necessary calls (likely set_ip_config and add_firewall_rules) using the information from Nexus.

Update

@rzezeski had a discussion about "should OPTE be notified for updated data (push model)" or "should OPTE be requesting data, so it accesses fresh copies (pull model)"?

Section 8 of RFD 63 suggests that there exists:

A control plane oriented agent that handles things like:

• Asking the control plane for the appropriate transformations for an instance and updating them as appropriate.
• Handling non-trivial actions such as translating DNS and metadata requests into the corresponding control plane calls.

Although that phrasing suggests some sort of mechanisms for making requests from OPTE -> The broader control plane, we're going to aim for an initial integration where control messages flow the opposite direction (namely, from the Sled Agent -> OPTE).

When VPC, Subnet, or Network interface objects are updated in Nexus, we'll need to perform a join-based query in the database to identify:

• Given a particular network resource being updated (VPC, subnet, whatever) ...
• ... what NetworkInterface objects are a part of this network...
• ... which Instance objects are they attached to...
• ... and for those Instance objects, which Sleds are they on?

Nexus can begin sending request to each sled, updating the live representation of all NetworkInterface objects. These will ultimately result in calls made to the OpteAdm object held by the SledAgent (mentioned in the "Initialization" section of this issue) for each interface. This process could potentially be updated to "one request per sled" if the Sled Agent held a mapping of "VPC -> Vnic" and "Subnet -> Vnic", and could internally identify all NICs needing updates after seeing a request to update a VPC/Subnet.

[smklein]

FYI - had a follow-up chat with @rmustacc to confirm the intent of RFD 63.

• The "pull model" for updates is appropriate for unbounded sets, like DNS names. We do not control all DNS names, so a push model would be unfeasible here.
• The "push model" for updates is appropriate for bounded sets, and routing info + firewalls fall into this category.

In summary, the approach of Nexus -> Sled Agent -> OPTE data flow (rather than any "upcall") will be sufficient until we need to tackle DNS lookups. At that point, we'll probably need to have a richer API for OPTE to access the Sled Agent itself, but we can wait on figuring that out.

[rmustacc]

The "pull model" for updates is appropriate for unbounded sets, like DNS names. We do not control all DNS names, so a push model would be unfeasible here.

I think would I clarify that these aren't updates, but rather us getting requests from VMs.

[rzezeski]

In the long-term, this allocation should likely be replaced with an API to directly allocate VNICs from OPTE.

I believe the goal is to eventually have OPTE control the allocation of the interfaces for the guest. Today that's a VNIC, created via dladm, overtop of a single link/NIC. However, the end goal is ECMP over two physical NICs. There is no support for this in VNICs (here I'm talking about illumos VNICs, not the general notion of a VNIC). Instead, I imagine that sled agent will make some ioctl call to the OPTE control node (e.g. /dev/opte) to create a new "port". The reason I refer to it as a port and not a VNIC is that what we will probably end up with is a virtual switch inside OPTE itself. This switch will have each NIC attached as its own upstream port, and each guest interface/VNIC as its own port. When creating a guest port, the sled agent will have to provide concrete details like the MAC address (e.g. a8:40:25:00:00:12), the IPv4/IPv6 of the guest interface (e.g. 192.168.2.18), and the VPC subnet (e.g. 192.168.2.0/24). Those are the least amount of things needed, we might also want other stuff like outbound NAT resources, a particular mac/IP for the "gateway" (in reality OPTE is the GW but we might want its virtualization to be configurable), etc. Or perhaps that stuff is pushed on after initial creation. The point is: some ioctl, called by sled agent via a library crate like libopte, will create this port and configure the OPTE engine as needed.

In the short-term, we'll continue allocating VNICs via the current mechanism (dladm create-vnic), but we'll use the ioctl-based interface from OPTE to perform operations like setting IP configuration and firewall rules.

Yep, in the short term I imagine it will look like this:

1. Sled Agent creates the VNIC.
2. Sled Agent either opens a new handle (or has one stashed) to OpteAdm (handle to control node at /dev/opte) and sends a request to register a new port, using the same name as the VNIC. This creates the requisite internal client instance in OPTE (OpteClientState) and creates a new minor node thru which Sled Agent can administer it.
3. Sled Agent opens an OptePort handle which provides access to the minor node representing this port.
4. Sled Agent sets the initial configuration of the port via OptePort (note that I might move this into step 2 as it could make the code cleaner, I won't know until I dive in).
5. Seld Agent will do the rest of its normal VM initializtion.
6. Any future requests, like adding/removing a firewall rule, can either open a new OptePort handle, or can stash the one from step 3 for future use (it really doesn't matter, it's just an open file descriptor).

it doesn't really work for shoving data into OPTE - at the point that this info is ready to be thrown at OPTE, it needs to be expanded

Precisely, at this point all data must be reified. OPTE, as much as possible, will not understand higher-level abstractions, and instead deals in specifics. That said, I could see some higher-level information being sent to OPTE for purposes of passing along in its packet. E.g., perhaps the interface (or entire instance) has a tag assigned to it in Nexus/Console, and that tag should be present on all packets leaving the guest. Sled Agent could call some API in libopte that sets the tag and then OPTE could include this tag in the Geneve header. Or similarly, in Geneve we will have a VNI (Virtual Network Id) that identifies the VPC. We could also have Sled Agent tell OPTE the VPC name so that it can add an additional named VPC tag in the Geneve header for the purposes of debugging.

We also might feed some of this higher-level detail to OPTE (like UUIDs) just for the purposes of debugging and human-friendliness. But the point is OPTE will not know how to convert UUID 1234 to VPC Subnet 192.168.2.0/24 without being told that explicitly by Sled Agent.

[bnaecker]

I'm going to close this for now. OPTE is integrated pretty well at this point, and we have specific issues tracking feature enhancements where needed.