Revise reservoir calculations for sleds with larger DRAM

[askfongjojo]

Currently the reservoir size for a gimlet with 2 TiB DRAM is 2x of 1-TiB gimlet, i.e. 1748894744576 bytes (~= 1.63 TiB, as seen in our dublin lab environment). This may be too conservative and considered a waste of resources by customers. A new calculation method (e.g., fixed-plus-variable) may be required to address the 2-TiB DRAM and future new HW situations.

Besides the large DRAM gimlet use case, it is also possible that we'll revise the calculations after profiling control plane usage further in the future. It will be desirable to allow the reservoir settings to be dynamic enough to accommodate such changes.

[askfongjojo]

Initial response from @gjcolombo:

It's probably needlessly conservative. If I've done my algebra correctly, we could go to about an 88.5% reservoir on this SKU while maintaining the same amount of physical memory that's not in the reservoir or specifically earmarked for OS usage (i.e. the page_t database). (I came up with this percentage by solving (0.2 * 1024) - 30 = ((1 - r) * 2048) - 60 for r.)

I expect most control plane services' charged memory usage not to depend much on the amount of physical memory in the machine, so unless we're changing other bits of configuration, most of the figures in https://rfd.shared.oxide.computer/rfd/0413#possible_budgets will apply to a 2 TiB Gimlet. The exceptions are:

• the page_t database will be 60 GiB instead of 30 (accounted for above)
• I'm unsure whether the APCB issue that prevented all physical memory from being addressed on the 1 TiB Gimlet is addressed (third row in the table)
• the Crucible numbers in the table are badly outdated and need to be recaptured (and ideally more rigorously)

There's also the general caveat that we haven't implemented any of RFD 413 yet, so we can't be certain the other figures in the table are especially accurate. But if they are accurate I wouldn't expect changing the physical memory size to change them.

[askfongjojo]

I can think of another argument for having some kind of background tasks to set the reservoir based on the current availability. In the event of DIMM failures, the system should not attempt to provision more VMMs that could lead to over-subscription. Crucially, the reservoir size change will not affect running instances (they may already be in a degraded state regardless); the goal of dynamically changing the reservoir to reflect the current capacity is to avoid making things worse.

[rmustacc]

I can think of another argument for having some kind of background tasks to set the reservoir based on the current availability. In the event of DIMM failures, the system should not attempt to provision more VMMs that could lead to over-subscription. Crucially, the reservoir size change will not affect running instances (they may already be in a degraded state regardless); the goal of dynamically changing the reservoir to reflect the current capacity is to avoid making things worse.

I think an initial reservoir size is important though. While background adjustment is useful, the longer the system is up, the more fragmented the pages will be and the harder it'll be to get contiguous memory to VMs. So it is important that we actually do get a large initial chunk into the reservoir.

[jclulow]

I'm not sure that we make use of available large page style contiguity today, FWIW. It's more about the cost of clawing it back.