API for the current rack utilization

[benjaminleonard]

As I understand it, presently if you want to look up the current utilization you need to provide a time range and grab the most recent entry.

As metrics are added when updated there's no guarantee that your requested time range contains any entries at all, so you need to set a start_time and end_time suitably far apart.

When we resolve how much capacity to reserve, that should also be included in the response.

[askfongjojo]

In the attempt to converge the many conversations on this topic, I'd propose that the API uses the allocated resources tracked in CRDB

1. Total provisionable capacity (the denominators)

Multiply the following "raw" capacity numbers by the estimated % of reserved capacity for OS and control plane, with a certain "fudge factor" on top as the safety net for under-estimation:

table	column(s)
sled	usable_hardware_threads, usable_physical_ram
zpool	total_size

For FCS, in discussion with @sdtuck, we'd like a number more conservative than 80% (the one that has been thrown around as a somewhat reasonable estimate) but defensible enough as system overhead. The API can use a set of fixed estimates for now and, over time, be replaced by more accurate estimates calculated programmatically based on real-time, self-reported statistics by sleds.

To conclude, we can use 70% as a start for CPU and memory and disk provisionable capacity. This multiplier should be made separate for CPU, memory, and disks since they will likely be different numbers in the future.

2. Current allocated capacity (the numerators)

All instances, regardless of their state, should be included in the math. The principle is again being more conservative initially as we continue to profile and tune system utilization. We can always revise the calculations as we introduce mechanisms to automatically reap failed instances and leverage resources from non-running instances in the future.

[askfongjojo]

@zephraph / @david-crespo - Please review what I wrote above and see if there is any gap or clarification that needs to be resolved for implementation.

[just-be-dev]

Okay, so from a base implementation perspective when we're looking at the latest moment in time of utilization (by which I mean what resources are provisioned) that all does feel like something we'd get directly from CRDB. We have to collect that kind of information anyway to make internal decisions.

I'm going to write out a basic view of what I understand here in explanation form which could be incorrect or incomplete.

The hard thing when we're talking about capacity and utilization is that there's a division between physical capacity and utilization and virtual capacity and utilization. If we allocate a virtual disk of 10 GiB the physical allocation could approximate to 30 GiB due to replication. From my understanding, we can't reliably map physical capacity to virtual capacity or vice versa. Then there's also the underlying notion that control plane software is running on the same hardware so it uses some level of physical capacity which intrinsically reduces the virtual capacity (which we don't have hard numbers on yet... there-in the resource padding @askfongjojo and @sdtuck suggest)

For customers deploying VMs to the rack they'll only ever see/know about virtual capacity. Operators will have deeper insights into the physical utilization of the rack but there's been some contention on how much to expose there based on how actionable that data is.

We can potentially do a very rough approximation based on the metrics pointed out by @askfongjojo with some multiplier for disk allocation but this still could be pretty wrong. I would typically lean on Sean or Robert here to flesh it out more.

Last thing:

All instances, regardless of their state, should be included in the math.

I don't think that part should be true. From my understanding a stopped instance uses no CPU/RAM (though its disks continue to be allocated). We would just use the same capacity checks for starting an instance as we would for creating one.

@askfongjojo explained on chat the desire for us to avoid over provisioning. That was actually outlined way back in RFD-70.

[benjaminleonard]

My view is that we should be displaying actual physical capacity and utilization. At least in the context of the capacity and utilization page.

For customers deploying VMs to the rack they'll only ever see/know about virtual capacity

I'm not sure this is correct, or rather we haven't explored presenting that information to the developer or silo/project admin. I think if we're showing virtual capacity to the people deploying VMs it should be in the context of (eventually) project limits. It's at this point that I think we'll want to figure out some way of approximately mapping virtual to physical utilization.

[just-be-dev]

My view is that we should be displaying actual physical capacity and utilization

For the developer inside of a silo the virtual cap & util is the actual. That's all they'll have access to. All they need to know is how many more CPUs / RAM / Disks can I allocate. For the operator the story is considerably more complex. We're defaulting to trying to only show what's actionable. Referring back to RFD-70 again:

Transparently show data. This does not mean show the customer the raw data, that would not be actionable. But frame the data in a way that is understandable. Make sure the data is clear, accurate, and actionable. The user can also export any data via the API, that way they can integrate easily into any existing systems they have.

While that RFD is older I think the point there still stands.

[askfongjojo]

I believe @sdtuck has indicated that provisionable capacity is what we want to display (at least initially). Physical capacity is a slippery slope and we can always deal with that as part of the customer walkthrough during rack standup and show them the numbers in the system hardware API. But for practical purposes, they should focus on the provisionable/virtual capacity which is more visible and the front-and-center thing in the console.

[askfongjojo]

We can potentially do a very rough approximation based on the metrics pointed out by @askfongjojo with some multiplier for disk allocation but this still could be pretty wrong. I would typically lean on Sean or Robert here to flesh it out more.

Totally. In fact, I have forgotten about the 3x disk allocation for each volume and there are two (?) disks that aren't meant for customer workload. Pinging @leftwo and @jmpesp here to see what would be the more accurate formula to derive the virtual disk capacity from physical disk capacity (before applying the reservation %).

[jmpesp]

The discussion on #1782 is relevant here. "provisionable capacity" is a way more difficult thing to pin down than physical capacity. I could tell you that there's 200G of physical space available, but depending on the conditions that won't mean that another 200G disk could be created, nor could a snapshot of an existing 100G disk. We also don't want to explicitly allocate up to 100% of the zpool.

I'm not sure we can give an accurate formula for deriving virtual disk capacity though. I haven't really thought about it for a while. I don't have a good handle on crucible overheads yet (how many bytes of overhead per guest byte). In a private chat, Alan suggested 3.5x the disk size, which could be a good starting point, but it's important to know that the overhead varies over time too (from the little testing I did, related to the size of sqlite's write ahead log).

[david-crespo]

We don't need it to be accurate in the sense of close to the true number. In the short term, we just need to be confident it's less than the true number.

[david-crespo]

We now have a UI showing current utilization based on virtual resource allocations divided by an estimated virtual cap which is itself based on estimated physical caps with a fudge factor so we can be conservative now and free up more space later. So I am calling the basic need here satisfied.

The fix was to allow order=desc on existing metrics endpoints so we can pull the most recent single data point for each metric, which we treat as the current value:

• Add order argument to get metrics in desc order #3269

There are several other issues around capping and constraining usage at resource create time:

• Launching a mega instance kills neighbors #3286
• Attempting to create a VM with more than 32 vcpus brings nexus down #3212
• Fail to provision instances with 96Gb memory (failed to await http server / tcp connect error) #3417
• decide if we want to apply constraints to disk size, memory size #2415