Define a concept of Artifact Visibility

[jkoritzinsky]

Asset Visibility

As part of building the VMR, we've found that we need to produce assets in multiple jobs that we used to produce in only one job. We have talked about doing asset selection through a list of jobs or asset name patterns. This works when we're only producing the assets that we want to publish.

However, we've discovered that in the VMR, we also need to build some additional packages that we never want to ship that we are effectively forced to name like a shipping package (in particular, we need to build a crossgen2 and ilc for the build host machine). These host-machine packages aren't assets we want to ship as they are built using LKG assets and assets that aren't targeting our "portable" build setup.

To handle these cases, we'd have to be very careful with a priority list of verticals as it would need to consider the machines we build on (so macOS x64 and macOS arm64 ordering would have to be ordered such that the primary vertical matches our build machine).

We looked at different artifact types and determined we have 3 visibilities, separate from Shipping/NonShipping:

• Vertical: Used within a VMR vertical, not published in the merged manifest. Not allowed in non-VMR builds.
• Internal: Used within a single AzDo pipeline run. Published from a job up to AzDO, but excluded from BAR and darc publishing. Good for cross-job and cross-BuildPass -only assets.
• External (default): Regular publishing.

To double check:

• The right tests are in and the right validation has happened. Guidance: https://github.com/dotnet/arcade/blob/main/Documentation/Validation.md

Validation that the jobs are dropped before going to BAR and the dropped artifacts are not uploaded to feeds by darc:

WindowsDesktop job that published to BAR with all rid-agnostic assets and exe installers marked as "Internal": https://dev.azure.com/dnceng/internal/_build/results?buildId=2604238&view=results
BAR record for that job showing that no rid-agnostic packages nor exe installers are present: https://maestro.dot.net/channel/529/azdo:dnceng:internal:dotnet-windowsdesktop/build/250107
Build Promotion job that shows that the internal assets were excluded from upload (the job failure later is due to an unrelated ongoing FR issue): https://dev.azure.com/dnceng/internal/_build/results?buildId=2604256&view=logs&j=ba23343f-f710-5af9-782d-5bd26b102304&t=74531eb2-9b39-5603-839e-94e3ba212b65&l=278

[ViktorHofer]

LGTM aside from one comment about the code in the PublishArtifactsInManifestBase type.

[mmitche]

Can you add documentation for this in https://github.com/dotnet/arcade/blob/992d2e5d76cccb72ad611ba2141e25db86cd59bd/Documentation/ArcadeSdk.md?

[jkoritzinsky]

I've updated this PR to exclude the assets before they even get into the manifest, but after uploading. That should remove the need to account for it in various places.

[jkoritzinsky]

Added docs. This is ready for another round of review.