BCM: Emergency staging pipeline optimizations

[mmitche]

• This issue is blocking
• This issue is causing unreasonable pain

As part of the BCM effort, we need to be able to deliver a .NET fix as fast as possible. This is not for "normal" fixes or releases. The idea is to deliver a single fix as fast as possible, and thus some guardrails or verification could be removed to speed things up. One area where we spend a lot of time is the staging pipeline. As part of this work, we should add configurability to the staging pipeline (e.g. via an "emergency" switch), that would eliminate the following non-critical stages/jobs, or take them off the critical path and make them run in parallel.

• Job - Validate * Inputs and Outputs - Eliminate in all critical path jobs.
• Stage - Required Validation - Run in parallel (non critical path, doesn't gate anything)
• Stage - Source code validation - Run in parallel (non critical path, doesn't gate anything)
• Stage - Validaton Ring - Run in parallel (non critical path, doesn't gate anything)
• Stage - SBOM Generation Ring - Run in parallel (non-critical path, doesn't gate anything)

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

[tkapin]

@mmitche Is there a deadline by which this needs to be implemented? If not, what's the priority of this?

@michellemcdaniel - Can you help with estimating the cost of this? It doesn't seem to be a big change, but we need to test it diligently and I'm not sure how testable the staging pipeline is. From my perspective, this shouldn't take more than a few days (to a person who's not familiar with the pipeline).

[mmitche]

This is for project standard. We're supposed to be showing consistent progress towards an improved delivery time. I cannot remember the exact timeline off the top of my head. @rhbanda?

[michellemcdaniel]

From the sound of it, this seems like we just need to add a new parameters and some conditionals around some of the dependencies? It almost might be just easier to create a separate pipeline with the stages in the right order. I think that second option would take less time, simply due to the amount of debugging messing with the stage dotnet pipeline's conditionals so that both the original and the expedited version work properly. That's basically what we've done with Validate-DotNet and the Fast Insertion pipelines. If we went the second way, I think once all requirements are collected it would take about a week, given that we'd want to change some of the conditionals on some of the jobs too and we'd need to modify the release pipeline as well (assuming that we'd need to be running that after the fact).

I think the first way (modifying stage-dotnet in place) would take a minimum of two weeks, but knowing how often we end up with weird bugs after the fact, it would have a much longer tail.

[tkapin]

Creating a separate pipeline seems very reasonable (presuming we'll be able to reuse the existing building blocks from the original staging pipeline) as we wouldn't add more complexity to the already complexity staging pipeline. Thanks for the estimations @michellemcdaniel.

@rbhanda - could you please clarify on the timeline and deadlines of this? Thanks

[mmitche]

Creating a separate pipeline seems very reasonable (presuming we'll be able to reuse the existing building blocks from the original staging pipeline) as we wouldn't add more complexity to the already complexity staging pipeline. Thanks for the estimations @michellemcdaniel.

@rbhanda - could you please clarify on the timeline and deadlines of this? Thanks

Wouldn't that affect the consumers of the staging pipeline (release and source build) who are coded up to pull from DotNet-Stage?

I think conditionals are a better way to go for long term maintainability. If we create a new pipeline, some types of updates to Stage-DotNet (new stages and potentially jobs) will not transparently show up in the new pipeline. It will likely quickly rot over time. For sure there will be rot in both scenarios, but I think it will be faster with a new pipeline.

[mmitche]

@mmitche Is there a deadline by which this needs to be implemented? If not, what's the priority of this?

@michellemcdaniel - Can you help with estimating the cost of this? It doesn't seem to be a big change, but we need to test it diligently and I'm not sure how testable the staging pipeline is. From my perspective, this shouldn't take more than a few days (to a person who's not familiar with the pipeline).

@tkapin, @rbhanda is thinking that Preview 2 would be a good goal to have this work done by. Any possibility that this could be resourced by then?

[michellemcdaniel]

Wouldn't that affect the consumers of the staging pipeline (release and source build) who are coded up to pull from DotNet-Stage?

I think conditionals are a better way to go for long term maintainability. If we create a new pipeline, some types of updates to Stage-DotNet (new stages and potentially jobs) will not transparently show up in the new pipeline. It will likely quickly rot over time. For sure there will be rot in both scenarios, but I think it will be faster with a new pipeline.

Yes, we would also have to do a bunch of work on Release-DotNet as well to make the "separate pipeline" idea work properly.

It's entire stages that would need to be skipped, right? So long as none of the stages that produce other bits are being skipped, I think it's 100% feasible to just do the conditional idea and get it correct

[mmitche]

stages skipped or dependencies changed, and then the Validate* contract jobs skipped.