Post-deploy k8s job for non-schema migrations

[bjester]

Current behavior

Recently, we've been adding individual k8s jobs for running specific django management commands for migrating existing data in the database after the deploy has occurred. This has benefits since these data migrations could take a while, and aren't usually critical to the functionality of the site, like schema migrations. Even with critical data migrations, we sometimes deploy those ahead of releases where we'll actually require that the data migrations have ran. Over time, we're accumulating a lot of individual jobs for these one-off migration management commands, usually invoked through the Makefile.

Desired behavior

If we had a generalized mechanism that hooks into the flow post-deploy, we could reuse it for calling these commands and we wouldn't need to create new k8s jobs every time. To start, we could create a new k8s job like we normally do, and have it call a make target, i.e. deploy-migrate, that we could change the recipe of whenever we need to / deploy. Long term, maybe there's a way we could leverage django's migration mechanism which doesn't call these migrations with the other schema migrations (make migrate)

Value add

Less infra requirement, less lingering relics in GCP and in the codebase

Possible tradeoffs

If we go the simple way to start, we could accidentally re-run migration commands if we forget to change the recipe of the make target. That may be a reason to explore leveraging django's migration system, or some other package, which is aware of what migrations have occurred.

[bjester]

@aronasorman Thoughts?

cc @rtibbles who needs another k8s job for a data migration needed for the upcoming major release

[aronasorman]

I agree. This will reduce the need for one-off jobs. In fact, we can repurpose an existing Kubernetes job to speed up the process, then consolidate into that.

The main concern with this, though, is ensuring we don’t block the continuous delivery pipeline during runs. That’s why we originally went with the cronjob-triggered-manually approach. This lets us run outside of the CD run, which times out after 30 minutes if we’re not careful.

So far we have two patterns for studio jobs:

• Post-migration runs, like what you mentioned here. These presumably take a while.
• Actual cronjobs on a set schedule, e.g. garbage collection.

We can consolidate these into two make targets to start, although we’ll still need to run post-migration jobs outside the CD pipeline. We can then trigger the post-migration job to run asynchronously, from the CD pipeline. However, that will have to be accounted for during development of these post-migration runs.

[bjester]

However, that will have to be accounted for during development of these post-migration runs.

@aronasorman could you explain further? Everything else you've mentioned is inline with my thoughts on the goals for this approach

[DXCanas]

I can't speak for Aron directly, but based on my familiarity of the code:
A migration is actually 3 separate steps in CD:

1. Update the app image used by the migration
2. Trigger the start of the migration
3. Wait for the migration to complete before closing off the CD

This job would be really similar — the main difference is that Step 3 wouldn't happen. No reason to wait if there's no other steps in queue to run after that's completed.

You won't get any logs in the CD pipeline (gcloud build's interface), but that won't be any different from the current debug process for any other job (Gcloud Logging will still pick them up, kubectl logs commands will still work, Sentry might even still be hooked up depending on the implementation, etc.)

If my understanding of this is right, it might actually be a decent Team Cohack session for infra. Maybe we could have a quick huddle early next week to establish the facts but in my head:

this is basically a migration job, but instead of running the migration it's just running a make command, and it's happening at the tail end of the CD pipeline.

[DXCanas]

Does this sound alright @bjester?

Also, is there a make command we could run right now as a standin?

[bjester]

@DXCanas @aronasorman added the make target here #4037

[bjester]

This has been live for a while