Ensure green test runs in VS

@vlada-shubina

Issue Description

Running unit tests in VS leads to some test failures - this should not be a case. VS-centric developer workflow should not be neglected

Classes of problems to tackle:

Tests using ExecMSBuild fail on core: Unit test failures when run from VS #8313

(Reported by @vlada-shubina) Some BuildEnvironmentHelper_Tests fail on core (extra RID in path) - Replacing traits used to skip certain test cases with custom Theory and Fact attributes #8348

Shouldly.ShouldAssertException : BuildEnvironmentHelper.Instance.MSBuildToolsDirectory32
    should be
"C:\Users\jankrivanek\AppData\Local\Temp\lhywa2vr.vir\2dfce9d58278472ea474ec9359cd7edd\MSBuild\Current\Bin"
    but was
"C:\Users\jankrivanek\AppData\Local\Temp\lhywa2vr.vir\2dfce9d58278472ea474ec9359cd7edd\MSBuild\Current\Bin\amd64"

Tests marked with TestPlatforms.AnyUnix run on Windows (and can fail). Can we use Need ability to skip test programmatically xunit/xunit#2073 here? - Replacing traits used to skip certain test cases with custom Theory and Fact attributes #8348
Test marked with SkipOnTargetFramework run on excluded FW (and can fail) Can we use Need ability to skip test programmatically xunit/xunit#2073 here? - Replacing traits used to skip certain test cases with custom Theory and Fact attributes #8348

StringTool RetainsLastStringWithGivenHashCode: Fix .NET 3.5 InternableString.GetHashCode to match the full implementation #8340

Shouldly.ShouldAssertException : _cache.GetDebugInfo()
   should be
Microsoft.NET.StringTools.WeakStringCache+DebugInfo (-1703002807)
    but was
Microsoft.NET.StringTools.WeakStringCache+DebugInfo (-1703002806)

ProjectItemSpecTooLong (might be env setup on my machine) - related? MSBuild task tests fail when long-paths enabled #4247
ProjectGetterResultsInWindowsDriveEnumerationWarning on Win (might be path specific?) - fixed by Allow MSBuild tests to access root of drive #7330
LogWarningUponProjectInstanceCreationFromDriveEnumeratingContent & LogWindowsWarningUponProjectInstanceCreationFromDriveEnumeratingContent (env specific?) - Allow MSBuild tests to access root of drive #7330
EvaluationProfiler_Tests (all) - no repro. (might have been caused by the bug in not unsetting the reverted change waves in tests)
LogWindowsWarningUponBuildingProjectWithDriveEnumeration - likely just needs application of the following fix: Add ability to create temp mapped drive for integration tests #8366

#8328 as inspiration for proper conditional skipping of tests

Running unit tests in VS leads to some test failures

Just wanted to mention that some of these issues reproduce on the command line as well. For example, dotnet test in src\Build.OM.UnitTests on Windows happily runs Unix-only test cases such as LogUnixWarningUponBuildingProjectWithDriveEnumeration.

LogUnixWarningUponBuildingProjectWithDriveEnumeration

It seems Visual Studio ignores those attributes [PlatformSpecific(TestPlatforms.AnyUnix)], whereas CI build filter tests based on them. I had the same situation with BuildEnvironmentHelper_Tests today.

Visual Studio shows them as failing

but they are still run.

I'm not sure whether this is a problem in how VS handling traits, or this is expected UX in VS, imo custom theory / fact attributes seems better solution as they work in VS and cmd and CI equally.

Not related to VS Run, but when running test of x64: .\build.cmd -test -ci /p:Platform="x64
Microsoft.Build.Graph.UnitTests.IsolateProjectsTests.UndeclaredReferenceEnforcementShouldNormalizeFilePaths took more than 1:30, after i killed the job.

#8348 solves # 2, 3 and 4 in the list, and likely other cases as well. The Unix only cases will be skipped in VS now, as the tests running on unsupported framework. This should reduce number of red tests in VS.

Add ability to create temp mapped drive for integration tests (#8366)

Tests which are still failing:

Microsoft.Build.Utilities.UnitTests (net7.0): 1 test
Microsoft.Build.UnitTests.PrintLineDebugger_Tests.ArtifactsDirectoryLooksGood

Microsoft.Build.Utilities.UnitTests (net472): 1 test
Microsoft.Build.UnitTests.PrintLineDebugger_Tests.ArtifactsDirectoryLooksGood


Microsoft.Build.Tasks.UnitTests (net7.0): 13 tests + 1 hangs
Microsoft.Build.UnitTests.CreateItem_Tests.RecursiveDirOutOfProc - hangs
Microsoft.Build.UnitTests.PrintLineDebugger_Tests.ArtifactsDirectoryLooksGood
Microsoft.Build.UnitTests.MSBuildTask_Tests.ProjectItemSpecTooLong
Microsoft.Build.UnitTests.CreateItem_Tests.LogWindowsWarningUponItemCreationWithDriveEnumeration (3 sets)
Microsoft.Build.UnitTests.CreateItem_Tests.LogWindowsWarningUponCreateItemExecution (3 sets)
Microsoft.Build.Tasks.UnitTests.RoslynCodeTaskFactory_Tests.RoslynCodeTaskFactory_ReuseCompilation
Microsoft.Build.Tasks.UnitTests.ResourceManagement_Tests (4 tests)


Microsoft.Build.Tasks.UnitTests (net472): 8 tests
Microsoft.Build.UnitTests.PrintLineDebugger_Tests.ArtifactsDirectoryLooksGood
Microsoft.Build.UnitTests.MSBuildTask_Tests.ProjectItemSpecTooLong
Microsoft.Build.UnitTests.CreateItem_Tests.LogWindowsWarningUponItemCreationWithDriveEnumeration (3 sets)
Microsoft.Build.UnitTests.CreateItem_Tests.LogWindowsWarningUponCreateItemExecution (3 sets)


Microsoft.Build.Engine.UnitTests (net7.0): 55 tests
Microsoft.Build.Engine.UnitTests.EvaluationProfiler_Tests (12 tests) - also took 60 minutes
Microsoft.Build.Engine.UnitTests.BackEnd.TaskHostFactory_Tests (all tests) - 2 tests - took 10 minutes
Microsoft.Build.Engine.UnitTests.ProjectCache.ProjectCacheTests.RunningProxyBuildsOnOutOfProcNodesShouldIssueWarning - 2 sets took 10 minutes
Microsoft.Build.Engine.UnitTests.ProjectCache.ProjectCacheTests.ProxyCacheHitsOnPreviousCacheMissesShouldWork - took 40 minutes
Microsoft.Build.Engine.UnitTests.ProjectCache.ProjectCacheTests.ProjectCacheByVsScenarioWorks - took 40 minutes
Microsoft.Build.Engine.UnitTests.ProjectCache.ProjectCacheTests.ProjectCacheByBuildParametersAndGraphBuildWorks - took 30 minutes
Microsoft.Build.Engine.UnitTests.ProjectCache.ProjectCacheTests.ProjectCacheByBuildParametersAndBottomUpBuildWorks - took 35.1 minutes
Microsoft.Build.Engine.UnitTests.ProjectCache.ProjectCacheTests.ParallelStressTestForVsScenario - 2 sets - took 160.1 minutes
Microsoft.Build.Engine.UnitTests.ProjectCache.ProjectCacheTests.ParallelStressTest (1 data set) - took 20 minutes
Microsoft.Build.Engine.UnitTests.ProjectCache.ProjectCacheTests.CacheShouldNotGetQueriedForNestedBuildRequests - took 5 minutes
Microsoft.Build.Engine.UnitTests.ProjectCache.ProjectCacheTests.CacheShouldBeQueriedInParallelDuringGraphBuilds - 1 set - took 10 minutes
Microsoft.Build.Graph.UnitTests.IsolateProjectsTests.UndeclaredReferenceEnforcementShouldNormalizeFilePaths - 70 minutes
Microsoft.Build.Graph.UnitTests.IsolateProjectsTests.UndeclaredReferenceEnforcementShouldFailOnUndeclaredReference - 2 sets -  10 minutes
Microsoft.Build.Graph.UnitTests.IsolateProjectsTests.UndeclaredReferenceEnforcementShouldFailOnPreviouslyBuiltButUndeclaredReferences - 5 minutes
Microsoft.Build.Graph.UnitTests.IsolateProjectsTests.SkippedTargetsShouldNotTriggerCacheMissEnforcement
Microsoft.Build.Graph.UnitTests.IsolateProjectsTests.IsolationRelatedMessagesShouldNotBePresentInNonIsolatedBuilds
Microsoft.Build.Graph.UnitTests.IsolateProjectsTests.IsolationRelatedMessageShouldBePresentInIsolatedBuildsWithMessaging - 5 mintes
Microsoft.Build.Graph.UnitTests.IsolateProjectsTests.EnforcementsCanBeSkipped - 4 sets - 20 minutes
Microsoft.Build.Graph.UnitTests.IsolateProjectsTests.CacheEnforcementShouldFailWhenReferenceWasNotPreviouslyBuiltWithoutContinueOnError
Microsoft.Build.Graph.UnitTests.IsolateProjectsTests.CacheEnforcementShouldAcceptPreviouslyBuiltReferences - 2 sets - 10 minutes
Microsoft.Build.Graph.UnitTests.IsolateProjectsTests.CacheAndUndeclaredReferenceEnforcementShouldAcceptSelfReferences - 2 sets - 10 minutes
Microsoft.Build.Graph.UnitTests.IsolateProjectsTests.CacheAndUndeclaredReferenceEnforcementShouldAcceptCallTarget
Microsoft.Build.UnitTests.EscapingInProjects_Tests.SimpleScenarios.SemicolonInPropertyPassedIntoStringParam_UsingTaskHost
Microsoft.Build.UnitTests.EscapingInProjects_Tests.SimpleScenarios.ItemTransformContainingSemicolon_InTaskHost
Microsoft.Build.UnitTests.EscapingInProjects_Tests.SimpleScenarios.EscapedWildcardsShouldNotBeExpanded_InTaskHost
Microsoft.Build.UnitTests.PrintLineDebugger_Tests.ArtifactsDirectoryLooksGood
Microsoft.Build.UnitTests.MSBuildTask_Tests.ProjectItemSpecTooLong
Microsoft.Build.UnitTests.FileMatcherTest.DriveEnumeratingWildcardIsLoggedOnWindows (4 sets)
Microsoft.Build.UnitTests.BackEnd.TaskHost_Tests.TasksGetNoGlobalPropertiesIfNoneSpecified
Microsoft.Build.UnitTests.BackEnd.TaskHost_Tests.TasksCanGetGlobalProperties
Microsoft.Build.UnitTests.BackEnd.SdkResultOutOfProc_Tests.RunOutOfProcBuild - 5 minutes
Microsoft.Build.UnitTests.BackEnd.SdkResultOutOfProc_Tests.RunOutOfProcBuildWithTwoProjects - 5 minutes
Microsoft.Build.UnitTests.BackEnd.BuildManager_Tests.VerifyMultipleRequestForSameProjectWithErrors_Simple
Microsoft.Build.UnitTests.BackEnd.BuildManager_Tests.VerifyMultipleRequestForSameProjectWithErrors_OnErrorChain
Microsoft.Build.UnitTests.BackEnd.BuildManager_Tests.VerifyMultipleRequestForSameProjectWithErrors_ErrorAndContinue
Microsoft.Build.UnitTests.BackEnd.BuildManager_Tests.VerifyMultipleRequestForSameProjectWithErrors_DifferentEntrypoints
Microsoft.Build.UnitTests.BackEnd.BuildManager_Tests.TestSimultaneousSubmissionsWithLegacyThreadingData_P2P_MP - do not run in VS
Microsoft.Build.UnitTests.BackEnd.BuildManager_Tests.TaskInputLoggingIsExposedToTasks (2 data sets)
Microsoft.Build.UnitTests.BackEnd.BuildManager_Tests.SimpleP2PBuildOutOfProc
Microsoft.Build.UnitTests.BackEnd.BuildManager_Tests.SimpleP2PBuildInProc
Microsoft.Build.UnitTests.BackEnd.BuildManager_Tests.SimpleBuildWithFailure
Microsoft.Build.UnitTests.BackEnd.BuildManager_Tests.SimpleBuildOutOfProcess
Microsoft.Build.UnitTests.BackEnd.BuildManager_Tests.ShouldBuildMutatedProjectInstanceWhoseProjectWasPreviouslyBuiltAsAP2PDependency
Microsoft.Build.UnitTests.BackEnd.BuildManager_Tests.SequentialBuildsOfTheSameProjectAllowed
Microsoft.Build.UnitTests.BackEnd.BuildManager_Tests.SequentialBuild
Microsoft.Build.UnitTests.BackEnd.BuildManager_Tests.RequestedResultsAreSatisfied
Microsoft.Build.UnitTests.BackEnd.BuildManager_Tests.Regress265010
Microsoft.Build.UnitTests.BackEnd.BuildManager_Tests.Regress251333
Microsoft.Build.UnitTests.BackEnd.BuildManager_Tests.Regress239661
Microsoft.Build.UnitTests.BackEnd.BuildManager_Tests.ProjectInstanceTransfersToOOPNode
Microsoft.Build.UnitTests.BackEnd.BuildManager_Tests.OverlappingBuildSubmissions
Microsoft.Build.UnitTests.BackEnd.BuildManager_Tests.OutOfProcProjectInstanceBasedBuildDoesNotReloadFromDisk(2 data sets)
Microsoft.Build.UnitTests.BackEnd.BuildManager_Tests.OutOfProcFileBasedP2PBuildSucceeds
Microsoft.Build.UnitTests.BackEnd.BuildManager_Tests.OutOfProcEvaluationIdsUnique
Microsoft.Build.UnitTests.BackEnd.BuildManager_Tests.MultiProcReentrantProjectWithCallTargetDoesNotFail
Microsoft.Build.UnitTests.BackEnd.BuildManager_Tests.MsBuildForwardAllPropertiesFromChildLaunchChildNode
Microsoft.Build.UnitTests.BackEnd.BuildManager_Tests.GraphBuildFail
Microsoft.Build.UnitTests.BackEnd.BuildManager_Tests.ForwardNoPropertiesLaunchChildNodeDefault
Microsoft.Build.UnitTests.BackEnd.BuildManager_Tests.ForwardNoPropertiesLaunchChildNode
Microsoft.Build.UnitTests.BackEnd.BuildManager_Tests.FailedAfterTargetInP2PShouldCauseOverallBuildFailure_MultipleEntrypoints(1 data set)
Microsoft.Build.UnitTests.BackEnd.BuildManager_Tests.ExplicitInprocAffinityGetsOverruledByDisableInprocNode
Microsoft.Build.UnitTests.BackEnd.BuildManager_Tests.DisableInProcNode
Microsoft.Build.UnitTests.BackEnd.BuildManager_Tests.CancelledBuildWithDelay40
Microsoft.Build.UnitTests.BackEnd.BuildManager_Tests.CacheLifetime



Microsoft.Build.Engine.UnitTests (net472): 12 tests
Microsoft.Build.UnitTests.PrintLineDebugger_Tests.ArtifactsDirectoryLooksGood
Microsoft.Build.UnitTests.MSBuildTask_Tests.ProjectItemSpecTooLong
Microsoft.Build.UnitTests.FileUtilities_Tests.FileOrDirectoryExistsNoThrowTooLongWithDotsRelative
Microsoft.Build.UnitTests.FileUtilities_Tests.FileOrDirectoryExistsNoThrowTooLongWithDots
Microsoft.Build.UnitTests.FileUtilities_Tests.FileExistsNoThrowTooLongWithDotsRelative
Microsoft.Build.UnitTests.FileUtilities_Tests.FileExistsNoThrowTooLongWithDots
Microsoft.Build.UnitTests.FileUtilities_Tests.DirectoryExistsNoThrowTooLongWithDotsRelative
Microsoft.Build.UnitTests.FileMatcherTest.DriveEnumeratingWildcardIsLoggedOnWindows (4 sets)
Microsoft.Build.Engine.UnitTests.BuildEnvironmentHelper_Tests.BuildEnvironmentFindsAmd64RunningInAmd64


Microsoft.Build.Engine.OM.UnitTests (net7.0): 4 tests
Microsoft.Build.UnitTests.OM.Instance.ProjectItemInstance_Tests.LogWindowsWarningUponBuildingProjectWithDriveEnumeration (4 sets)

Microsoft.Build.Engine.OM.UnitTests (net472): 4 tests
Microsoft.Build.UnitTests.OM.Instance.ProjectItemInstance_Tests.LogWindowsWarningUponBuildingProjectWithDriveEnumeration (4 sets)


Microsoft.Build.CommandLine.UnitTests (net472): 1 test
Microsoft.Build.Engine.UnitTests.MSBuildServer_Tests.MSBuildServerTest

Related issue: #8457

Enable Windows Disabled Drive Enumeration Tests (#9266)

Enable Windows Disabled Drive Enumeration Tests (dotnet#9266)

Enable Windows Disabled Drive Enumeration Tests (#9266)

Navigation Menu

Repository navigation

Ensure green test runs in VS #8329

Issue Description

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Ensure green test runs in VS #8329

Description

Issue Description

Activity

JanKrivanek commented on Jan 23, 2023

ladipro commented on Jan 23, 2023

vlada-shubina commented on Jan 23, 2023

vlada-shubina commented on Jan 24, 2023

vlada-shubina commented on Jan 24, 2023

vlada-shubina commented on Feb 16, 2023

vlada-shubina commented on Feb 16, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions