Big refactor #2 + Jobs Jobs Jobs

[mariusandra]

Changes

• Should be merged after Big refactor #350 is merged, which should be merged after Retry queue #325 is merged
• Rename "retry queue" to "job queue"
• Refactor worker tasks (get rid of the huge if, replace it with a smaller object... one step at a time)
• Implements the following syntax for plugin jobs:

export const jobs = {
    retryProcessEvent: (event, meta) => {
        console.log('retrying event!', event.event)
    }
}
export async function processEvent (event, meta) {
    if (event.properties?.hi === 'ha') {
        console.log('processEvent')
        // queues to run in 30 seconds
        meta.jobs.retryProcessEvent(event).runIn(30, 'seconds')
        // queues to run async but now
        meta.jobs.retryProcessEvent(event).runNow()
    }
    return event
}

I'm also very happy to discuss changes to the API. This is the PR to do that.

Checklist

• Updated Settings section in README.md, if settings are affected
• Jest tests

[Twixes]

In the spirit of PostHog/plugin-scaffold#16 (review), could runIn not take target and key directly?

[mariusandra]

Hm... we don't really use target at all, so I assume you mean payload? I mean we can have:

job.runIn(3, 'minutes', 'taskName', { task: 'payload' })

... but doesn't really spark joy.

job.runIn(3, 'minutes').taskName({ task: 'payload' })

Not sure, it's all very subjective :). Feel free to make some plugin and play around with alternative APIs for this.

[mariusandra]

Alternatively, we can also flip it around:

jobs.taskName({ task: 'payload' }).runIn(3, 'minutes')
jobs.taskName({ task: 'payload' }).runNow()

[mariusandra]

All right, the last changes in this PR flip the functions around and now this is how it works:

export const jobs = {
    retryProcessEvent: (event, meta) => {
        console.log('retrying event!', event.event)
    }
}

export async function processEvent (event, { jobs }) {
    if (event.properties?.hi === 'ha') {
        console.log('processEvent')

        // await to be sure it got enqueued, otherwise skip it and YOLO

        await jobs.retryProcessEvent(event).runIn(30, 'seconds')
        await jobs.retryProcessEvent(event).runNow()
        await jobs.retryProcessEvent(event).runAt(new Date())
    }
    return event
}

@yakkomajuri @Twixes would this syntax work for you? In case tests pass (this PR includes some flake fixes and other cleanup), I might merge this anyway and we can always tweak it later this week.

[yakkomajuri]

Quite like the new syntax - also happy to merge and iterate if necessary

[mariusandra]

Just tested this with Amazon Aurora Serverless (thanks @fuziontech) and it actually works with it! 🎉
We should run some experimental load tests to see what it can handle.

In any case, I'd still like to improve the layering and fallback system for these job queues. So plenty of refactoring still to come.

[Twixes]

Should this be in here, as just hello? Seems a bit cryptic/odd

[mariusandra]

That can be most likely deleted. I just copied this from the previous place. Next time, as there are more refactors to come.

[Twixes]

That flipped API looks pretty great 👌 @mariusandra