PerfView & PerfViewCollect not working in a Windows Server Core k8s pod

[ysiivan]

Sysinfo

WindowsBuildLabEx                                       : 18362.1.amd64fre.19h1_release.190318-1202
WindowsCurrentVersion                                   : 6.3
WindowsEditionId                                        : ServerDatacenterACor
WindowsInstallationType                                 : Server Core
WindowsInstallDateFromRegistry                          : 8/6/2020 4:28:25 PM
WindowsProductId                                        : 00431-40000-00000-AA301
WindowsProductName                                      : Windows Server Datacenter
WindowsRegisteredOrganization                           :
WindowsRegisteredOwner                                  :
WindowsSystemRoot                                       : C:\Windows
WindowsVersion                                          : 1903
BiosCharacteristics                                     : {4, 7, 9, 11...}
BiosBIOSVersion                                         : {INTEL  - 6040000, VMW71.00V.17369862.B64.2012240522,
                                                          VMware, Inc. - 10000}

Problem (PerfView)

[EXECUTING: PerfView /logFile=log.txt /maxCollectSec=60 collect]
Circular buffer size = 0, setting to 500.
[Kernel Log: C:\diag\output\PerfViewData.kernel.etl]
Kernel keywords enabled: Default
Aborting tracing for sessions 'NT Kernel Logger' and 'PerfViewSession'.
Ensuring .NET Allocation profiler not installed.
Detected 64 bit system, removing 32 bit keys.
[Exception Occurred: System.Runtime.InteropServices.COMException (0x80070522): A required privilege is not held by the c
lient. (Exception from HRESULT: 0x80070522)
   at System.Runtime.InteropServices.Marshal.ThrowExceptionForHRInternal(Int32 errorCode, IntPtr errorInfo)
   at Microsoft.Diagnostics.Tracing.Session.TraceEventSession.EnableKernelProvider(Keywords flags, Keywords stackCapture
)
   at PerfView.CommandProcessor.Start(CommandLineArgs parsedArgs)
   at PerfView.CommandProcessor.Collect(CommandLineArgs parsedArgs)
   at PerfView.CommandProcessor.ExecuteCommand(CommandLineArgs parsedArgs)]
[DONE 16:16:00 FAIL: PerfView /logFile=log.txt /maxCollectSec=60 collect]

Problem (PerfViewCollect)

PS C:\diag\output> ..\pvc\PerfViewCollect.exe /MaxCollectSec:30 /LogFile:log.txt collect

Pre V4.0 .NET Rundown disabled, Type 'E' to enable symbols for V3.5 processes.
Type S to stop collection, 'A' will abort.
PS C:\diag\output> cat .\log.txt
PerfView logging started at 1/18/2023 5:21:22 PM
[EXECUTING: PerfView /MaxCollectSec:30 /LogFile:log.txt collect]
Circular buffer size = 0, setting to 500.
[Kernel Log: C:\diag\output\PerfViewData.kernel.etl]
Kernel keywords enabled: Default
Aborting tracing for sessions 'NT Kernel Logger' and 'PerfViewSession'.
Ensuring .NET Allocation profiler not installed.
Detected 64 bit system, removing 32 bit keys.
[Exception Occurred: System.Runtime.InteropServices.COMException (0x80070522): A required privilege is not held by the c
lient. (0x80070522)
   at System.Runtime.InteropServices.Marshal.ThrowExceptionForHR(Int32 errorCode)
   at Microsoft.Diagnostics.Tracing.Session.TraceEventSession.EnableKernelProvider(Keywords flags, Keywords stackCapture
) in C:\work\apg\perfview\src\TraceEvent\TraceEventSession.cs:line 734
   at PerfView.CommandProcessor.Start(CommandLineArgs parsedArgs) in C:\work\apg\perfview\src\PerfView\CommandProcessor.
cs:line 502
   at PerfView.CommandProcessor.Collect(CommandLineArgs parsedArgs) in C:\work\apg\perfview\src\PerfView\CommandProcesso
r.cs:line 208
   at PerfView.CommandProcessor.ExecuteCommand(CommandLineArgs parsedArgs) in C:\work\apg\perfview\src\PerfView\CommandP
rocessor.cs:line 38]
[DONE 17:21:22 FAIL: PerfView /MaxCollectSec:30 /LogFile:log.txt collect]

User

PS C:\diag\output> whoami
user manager\containeradministrator
PS C:\diag\output>
PS C:\diag\output> $currentPrincipal = New-Object Security.Principal.WindowsPrincipal([Security.Principal.WindowsIdentit
y]::GetCurrent())
PS C:\diag\output> $currentPrincipal.IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator)
True

[brianrob]

Are you using process isolation containers? If so, this error is expected, because the kernel is shared amongst all of the containers on the machine, and allowing those events to flow into the container would leak information from outside the container.

The recommendation here is to capture a trace on the host with /EnableEventsInContainers, and then copy the trace into the container and run PerfView merge -ImageIDsOnly against the trace. This will ensure that you get kernel events, and also that your stacks are resolvable.

[brianrob]

Closing per recommendation above.

[Cjewett]

Hi @brianrob. I'm currently running into a similar issue trying to run PerfView in a k8s pod. For those less knowledgeable with this tool is it possible to provide more context of what to do in this scenario?

Is the expectation on the Host node a user manually starts the PerfView process to collect a trace using the /EnbleEventsInContainer parameter. After they are done collecting the trace you then copy the zip into the container and perform a PerfView merge -ImageIDsOnly NameOfTrace.etl.zip? What does -ImageIDsOnly do exactly? I run that command and can't really tell if it worked.

If you don't run -ImageIDsOnly is it still possible to use that trace to profile what occurred in the k8s pod?

If this should be a new issue or discussion please let me know. Thanks!

[brianrob]

@Cjewett, not sure if you got a notification. In case you didn't - see #2343.