Linux VM disables PMU EL0 by default

[terryzbai]

After Linux VM boots, PMU counters are not accessible to userspace applications. My hacky solution was re-enabling it via a custom system call at very beginning of notified function in VMM. This needs to patch the system call to seL4 kernel and recompile the Microkit SDK. It's necessary to figure out how to configure PMU access at VM's boot time or build time.

[midnightveil]

Is "userspace" here mean "linux userspace (guest userspace)" or does it mean non-VM seL4 native threads?

[terryzbai]

Both will lose the access to PMU counters

[Ivan-Velickovic]

In the past I've commented out this part in Linux:
https://github.com/torvalds/linux/blob/028ef9c96e96197026887c0f092424679298aae8/arch/arm64/include/asm/assembler.h#L487-L498

I guess the problem is that pmuserenr_el0 is allowed to be accessed by VMs running in EL1.

There's issues open in seL4 (e.g seL4/seL4#1018) about not letting direct access to these kinds of registers.

[Indanz]

I guess the problem is that pmuserenr_el0 is allowed to be accessed by VMs running in EL1.

I assume that the option KernelArmExportPMUUser is enabled and that all tasks are given access to the PMU registers, including VMs.

If so, you get what you asked for. It is up to user space to not step on each other's toes, as PMU registers are not context switched. Same applies to user space access to timer registers and other such settings for registers that are not context switched.

If KernelArmExportPMUUser is not enabled, the VM should not have access to the PMU registers either, otherwise that's a bug.

There's issues open in seL4 (e.g seL4/seL4#1018) about not letting direct access to these kinds of registers.

That issue does not apply to registers that are not context switched and are enabled with options like KernelArmExportPMUUser. For such registers the rules for EL1 tasks is the same as for EL0. It would be weird to add context switching for such registers for VCPUs, but not for EL0 tasks, as that would already be a policy decision.

It would make more sense to context switch the PMU cfg register if KernelArmExportPMUUser is enabled so that the VMM can control whether a VM has access to the PMU. But the same is true for normal tasks, so all-in-al the solution is to make it a per task flag that is controlled in user space.

As for context switching PMU registers, either at all or only for VCPUs: That would make it impossible to measure over all system performance, so could interfere with what the user wants to measure.

[Ivan-Velickovic]

If so, you get what you asked for. It is up to user space to not step on each other's toes, as PMU registers are not context switched. Same applies to user space access to timer registers and other such settings for registers that are not context switched.

Yes, I meant to say that the 'problem' is that we (in debug/benchmark configurations) allow user-space to access PMU registers so the Linux VM is free to mess up other user-space programs that want to use the PMU.

That issue does not apply to registers that are not context switched and are enabled with options like KernelArmExportPMUUser. For such registers the rules for EL1 tasks is the same as for EL0. It would be weird to add context switching for such registers for VCPUs, but not for EL0 tasks, as that would already be a policy decision.

I got confused when I made that comment, apologies. Ignore that part of my comment.

It would make more sense to context switch the PMU cfg register if KernelArmExportPMUUser is enabled so that the VMM can control whether a VM has access to the PMU. But the same is true for normal tasks, so all-in-al the solution is to make it a per task flag that is controlled in user space.

Yes I agree, that would be the solution.

[Indanz]

Yes, I meant to say that the 'problem' is that we (in debug/benchmark configurations) allow user-space to access PMU registers so the Linux VM is free to mess up other user-space programs that want to use the PMU.

I'd say that's fine, because this only happens if user space counts on PMU access, in which case it is a configuration error to have a Linux VM running on the same core as those PMU using tasks.

I agree it's all a bit annoying if you just want to use PMU just for cycle count measurements while there happens to be a Linux guest present.

Maybe we should set HCR_EL2.TID3 = 1 to trap reads of ID_AA64DFR0_EL1? Then VMMs can return what they want.