Enable source-build pre-built detection

[oleksandr-didyk]

Part of dotnet/source-build#3017

Enable source-build pre-build detection on the current repository and resolve any pre-build issues discovered

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[oleksandr-didyk]

As part of work on the task we discovered that in the context of repo build runtime's source-build is utilizing older versions of several components, specifically:

• Microsoft.CodeAnalysis: 4.4.0; example of a source-built project that is using this version -> Microsoft.Interop.SourceGeneration
• Microsoft.Build: 17.3.2; example of a source-built project that is using this version -> MonoAOTCompiler

In the context of current product source-build, these dependencies will be replaced by their respective latest versions. This creates a sizable difference in behaviour between the product source-build and the repo source-build.

@ViktorHofer pinging you here to get an opinion on what, in your mind, is a better option from runtime's perspective -> to have source-build continue using the latest versions (i.e. source-build differs from 'VS build') or for source-build to use the VS-compatible ones (i.e. source-build behaves the same as 'VS build')? The later will become possible with the introduction of repository ProjectVersions.props files.

CC: @MichaelSimons @mmitche

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Part of dotnet/source-build#3017

Enable source-build pre-build detection on the current repository and resolve any pre-build issues discovered

Author:	oleksandr-didyk
Assignees:	oleksandr-didyk
Labels:	area-Infrastructure, untriaged
Milestone:	-

[mmitche]

As part of work on the task we discovered that in the context of repo build runtime's source-build is utilizing older versions of several components, specifically:

• Microsoft.CodeAnalysis: 4.4.0; example of a source-built project that is using this version -> Microsoft.Interop.SourceGeneration
• Microsoft.Build: 17.3.2; example of a source-built project that is using this version -> MonoAOTCompiler

In the context of current product source-build, these dependencies will be replaced by their respective latest versions. This creates a sizable difference in behaviour between the product source-build and the repo source-build.

@ViktorHofer pinging you here to get an opinion on what, in your mind, is a better option from runtime's perspective -> to have source-build continue using the latest versions (i.e. source-build differs from 'VS build') or for source-build to use the VS-compatible ones (i.e. source-build behaves the same as 'VS build')? The later will become possible with the introduction of repository ProjectVersions.props files.

CC: @MichaelSimons @mmitche

This changed last week. Runtime now builds very early in the stack, mirroring the way the product builds. Therefore roslyn and msbuild do not flow into runtime.

I think the cases above are textbook examples of SBRPs.

[oleksandr-didyk]

Pre-builts currently getting removed / excluded:

• Microsoft.CodeAnalysis 4.4.0 -> getting bumped to latest VS version by Move all of the ref-pack generators to use the "LatestVS" Roslyn build so they don't pull in prebuilts in source-build #81561. After the PR is merged we'll check the status of the pre-build.
• System.Reflection.Metadata 6.0.1 -> added to SBRP add System.Reflection.Metadata 6.0.1 source-build-assets#506
• Microsoft.NET.ILLink.Tasks, Microsoft.NETCore.App.Crossgen2 and Microsoft.NETCore.App.Runtime -> would be marked as allowed pre-builts Dealing with pre-builts brought in by SDK bundled versions in repo source-build source-build#3228
• Microsoft.NETFramework.ReferenceAssemblies -> would be removed once TFM filtering is enabled Introduce targets to filter target frameworks arcade#12310

Pre-builts that ideally would require a version bump. @ViktorHofer would be ideal if we could know which of these can be updated (we would create separate issues for the update + allow the pre-builts in the infra until the said issue would get attention from runtime) and which for whatever reason cannot:

• Microsoft.Build 17.3.2 -> while creating SBRPs for the package and its dependencies would not be a hard process, this might create additional maintenance for the runtime team once the version will need a bump. From Git Blame, it seems that the last update to the package was a while back and was not related to any specific task - Update a few dependencies #77678
• Nuget.ProjectModel 6.2.2
• Microsoft.Extensions.DependencyModel 6.0.0
• System.CommandLine 2.0.0-beta4.22355.1 -> from trying out the latest version, it seems that for this package to be bumped source changes are required, as the package's API changed

[ViktorHofer]

Microsoft.NETFramework.ReferenceAssemblies -> would be removed once TFM filtering is enabled dotnet/arcade#12310

We already have our own infrastructure to exclude these if that's what we want. Do we have a tracking issue for that? I wasn't aware that we want to remove .NET Framework pre-builts right now. What's that issue's priority?

Microsoft.Build 17.3.2
Nuget.ProjectModel 6.2.2

AFAIK we are free to use any version here, i.e. the latest version from nuget.org. cc @dotnet/runtime-infrastructure
In the long-term, how do we keep these up-to-date and in sync with the rest of the VMR?

Microsoft.Extensions.DependencyModel 6.0.0

We build that library ourselves in the repository so we should remove the prebuilt package dependency entirely.

System.CommandLine 2.0.0-beta4.22355.1 -> from trying out the latest version, it seems that for this package to be bumped source changes are required, as the package's API changed

cc @adamsitnik

[oleksandr-didyk]

Microsoft.NETFramework.ReferenceAssemblies -> would be removed once TFM filtering is enabled dotnet/arcade#12310

We already have our own infrastructure to exclude these if that's what we want. Do we have a tracking issue for that? I wasn't aware that we want to remove .NET Framework pre-builts right now. What's that issue's priority?

We don't have a tracking issue just yet, since it would depend on when the changes would be usable in runtime. IIRC we would allow them as pre-builts if need be and remove them once the filtering is available and enabled.

Microsoft.Build 17.3.2
Nuget.ProjectModel 6.2.2

AFAIK we are free to use any version here, i.e. the latest version from nuget.org. cc @dotnet/runtime-infrastructure In the long-term, how do we keep these up-to-date and in sync with the rest of the VMR?

If we do bump them to latest - Maestro dependency flow would update them automatically trough subscriptions that would be created as part of this task. Other repositories would get the packages from Maestro as well, so it should sync up

Microsoft.Extensions.DependencyModel 6.0.0

We build that library ourselves in the repository so we should remove the prebuilt package dependency entirely.

Sounds great, I will create a tracking issue for it + add a comment to remove the allowed pre-built as part of that work

[ViktorHofer]

We don't have a tracking issue just yet, since it would depend on when the changes would be usable in runtime. IIRC we would allow them as pre-builts if need be and remove them once the filtering is available and enabled.

As said, TFM filtering IS already available in dotnet/runtime. Nothing blocks us from removing .NET Framework nodes from the source-build graph. If this is desirable and important, please file an issue for that (including the priority for the source build team).

through subscriptions that would be created as part of this task

Sounds like a SRBP source build subscription. Is that right?

Sounds great, I will create a tracking issue for it + add a comment to remove the allowed pre-built as part of that work

Thanks 👍

[oleksandr-didyk]

We don't have a tracking issue just yet, since it would depend on when the changes would be usable in runtime. IIRC we would allow them as pre-builts if need be and remove them once the filtering is available and enabled.

As said, TFM filtering IS already available in dotnet/runtime. Nothing blocks us from removing .NET Framework nodes from the source-build graph. If this is desirable and important, please file an issue for that (including the priority for the source build team).

My bad, I worded it badly. I was just providing context for why the issue wasn't raised. I will create and issue for it once I double-check its priority. Thanks for pointing this out

through subscriptions that would be created as part of this task

Sounds like a SRBP source build subscription. Is that right?

We would create subscriptions for the packages themselves, so for msbuild and nuget repos. The repos source-build their latest versions as part of their CI, which is then available for runtime's source-build restore from the *-transport NuGet feeds.

[ViktorHofer]

We would create subscriptions for the packages themselves, so for msbuild and nuget repos.

I thought we can't do that as runtime now builds before msbuild and nuget in the source-build graph? Above from @mmitche:

This changed last week. Runtime now builds very early in the stack, mirroring the way the product builds. Therefore roslyn and msbuild do not flow into runtime.