System.Diagnostics.Metrics.Tests.MetricEventSourceTests failing from event waiting timeouts

[hoyosjs]

Over the last 15 days, there's been a lot of outer loop failures for the metrics event source tests:

Method	Type	Count
EventSourcePublishesEndEventsOnNewListener	System.Diagnostics.Metrics.Tests.MetricEventSourceTests	236
EventSourcePublishesTimeSeriesWithMetadata	System.Diagnostics.Metrics.Tests.MetricEventSourceTests	236
EventSourcePublishesTimeSeriesForLateInstruments	System.Diagnostics.Metrics.Tests.MetricEventSourceTests	235
EventSourcePublishesTimeSeriesWithTags	System.Diagnostics.Metrics.Tests.MetricEventSourceTests	240
EventSourcePublishesTimeSeriesWithEmptyMetadata	System.Diagnostics.Metrics.Tests.MetricEventSourceTests	235
EventSourcePublishesEndEventsOnMeterDispose	System.Diagnostics.Metrics.Tests.MetricEventSourceTests	235
EventSourcePublishesAllDataTypes	System.Diagnostics.Metrics.Tests.MetricEventSourceTests	235
EventSourceEnforcesTimeSeriesLimit	System.Diagnostics.Metrics.Tests.MetricEventSourceTests	235
EventSourceHandlesObservableCallbackException	System.Diagnostics.Metrics.Tests.MetricEventSourceTests	235
EventSourceEnforcesHistogramLimit	System.Diagnostics.Metrics.Tests.MetricEventSourceTests	235
EventSourcePublishesMissingDataPoints	System.Diagnostics.Metrics.Tests.MetricEventSourceTests	235
EventSourcePublishesInstruments	System.Diagnostics.Metrics.Tests.MetricEventSourceTests	235
EventSourcePublishesTimeSeriesForLateMeter	System.Diagnostics.Metrics.Tests.MetricEventSourceTests	235

See this kusto query for queries to the full builds and logs.

cc: @noahfalk @tarekgh

[tarekgh]

@noahfalk could you please have a look? Thanks!

[tarekgh]

@hoyosjs couldn't execute the query. do you have a better one?

 'where' operator: Failed to resolve table or column expression named 'Jobs'

 clientRequestId: KustoWebV2;ecd650d0-fa01-4fa4-b660-ce9640bbb53e

[hoyosjs]

@tarekgh I opened the link again. Somehow it executes just fine. It needs to be done against the scope @engsrvprod/engineeringdata

[tarekgh]

Looks I am not authroized to execute it :-(

[hoyosjs]

System.Diagnostics.Metrics.Tests.MetricEventSourceTests.xlsx

[noahfalk]

It appears all the failures started 7/22 which was the exact day I checked in #56083 and I forgot to run the outerloop tests. Decent odds this change broke it. Current hypothesis is that the tests are unexpectedly running in parallel and hitting the error handler that intentionally blocks concurrent usage of the EventSource. I'm working on a code change to the tests to see if we can confirm this.

[noahfalk]

I think this PR fixes it: #56382

[hoyosjs]

Fixed in #56382