Race condition can cause big checklists

[roryabraham]

Problem

Sometimes we get big checklists like this one that include all the PRs from the previous checklist.

The root cause is a race condition between:

• createNewStagingDeployCash
• deployStaging

Both are triggered at the same time after updateStaging finishes. If deployStaging finishes (i.e: the tag for the new staging version is pushed to the remote) before the checklist is created, then it works. If not, then we get a huge checklist like this will all the PRs that were deployed in the previous version.

Solution

TBD – probably change the deploy system a bit such that staging deploys are triggered by the creation of a prerelease and production deploys by the creation of a "full" release.

That way we can push the tag as soon as it's created without triggering a deploy

[roryabraham]

No update right now

[roryabraham]

No update

[mountiny]

Looking into this one now

Noting that the createNewDeployChecklist was moved to the platformDeploy.yml, which feels like the race condition is not happening anymore and the bug might be somewhere else.

When the updateStaging finished, it force pushes to staging branch and then the deploy.yml runs which pushes the tags and finally we trigger the platformDeploy.yml which would create the checklist but the correct tags should be available

Looking more into why this is failing

[mountiny]

Created a PR which should hopefully help with this

[roryabraham]

Ok, it seems like there's some way the git history can change in production that's different from what we have represented in the CIGitLogicTest.sh, just judging by running this command on staging w/ full git history:

Neither is correct, because the only version-bump that we expect to be included between 1.3.76-6 and 1.3.77-0 is Update version to 1.3.77-0