As detailed in #1794, when an application is tracing itself there is the potential for a deadlock to occur if the reader tries to take the EventPipe configuration lock while EventPipe::Disable (specifically EventPipe::RunWithCallbackPostponed) is in progress.
This occurs when EventPipeEventSource is reading because the Trace Event library uses Regex in its logic and Regex now has a lazily initialized ConcurrentDictionary cache that generates events when used. This causes the static EventSource provider for concurrent collections to get created (unless its been instantiated earlier in the process) which will call into native code to create the provider. This requires the config lock in EventPipe. If this occurs during EventPipe::RunWithCallbackPostponed, you may deadlock because the reader is waiting on the lock, but the writer is holding it and is blocked because the pipe's buffer is full.
See the stacks in #1794 for details.
A couple potential paths to fixing this:
- break up the logic of
EventPipe::Disable so that it isn't holding the config lock for the entire duration. Rundown and other parts of disable can run for minutes in big applications, so this shouldn't be holding a global lock like that the entire time.
- find ways to defer the registration of new providers that get registered while
EventPipe::Disable is in flight.
This issue breaks our EventPipe tests, but #1794 added a bypass that needs to be removed once this issue is closed.
CC - @tommcdon @noahfalk @sywhang
As detailed in #1794, when an application is tracing itself there is the potential for a deadlock to occur if the reader tries to take the EventPipe configuration lock while
EventPipe::Disable(specificallyEventPipe::RunWithCallbackPostponed) is in progress.This occurs when
EventPipeEventSourceis reading because the Trace Event library uses Regex in its logic and Regex now has a lazily initializedConcurrentDictionarycache that generates events when used. This causes the staticEventSourceprovider for concurrent collections to get created (unless its been instantiated earlier in the process) which will call into native code to create the provider. This requires the config lock in EventPipe. If this occurs duringEventPipe::RunWithCallbackPostponed, you may deadlock because the reader is waiting on the lock, but the writer is holding it and is blocked because the pipe's buffer is full.See the stacks in #1794 for details.
A couple potential paths to fixing this:
EventPipe::Disableso that it isn't holding the config lock for the entire duration. Rundown and other parts of disable can run for minutes in big applications, so this shouldn't be holding a global lock like that the entire time.EventPipe::Disableis in flight.This issue breaks our EventPipe tests, but #1794 added a bypass that needs to be removed once this issue is closed.
CC - @tommcdon @noahfalk @sywhang