Skip to content
This repository was archived by the owner on Nov 4, 2021. It is now read-only.

S3 Job Queue - #399

Merged
mariusandra merged 9 commits into
masterfrom
new-job-queues
May 26, 2021
Merged

mariusandra merged 9 commits into
masterfrom
new-job-queues

Conversation

@mariusandra

@mariusandra mariusandra commented May 20, 2021 •

Copy link
Copy Markdown
Collaborator

Changes

  • Still WIP
  • Implements a simple redlocked job queue based on S3.
  • Needs tests
  • Closes S3 job queue #364

Checklist

  • Updated Settings section in README.md, if settings are affected
  • Jest tests

@yakkomajuri yakkomajuri mentioned this pull request May 20, 2021
@mariusandra
mariusandra marked this pull request as ready for review May 25, 2021 20:55
@mariusandra

Copy link
Copy Markdown
Collaborator Author

Here's something that could use a look. :)

I grouped the different job queues into folders based on functionality (local, concurrent, redlocked), but didn't yet implement different functionalities depending on the type. This means currently all job queues are still redlocked. I'd suggest tackling this in another PR.

@Twixes Twixes left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks reasonable to me

@mariusandra mariusandra added the bump patch Bump patch version when this PR gets merged label May 26, 2021
@mariusandra
mariusandra enabled auto-merge (squash) May 26, 2021 15:15
@mariusandra
mariusandra merged commit 4b72f4e into master May 26, 2021
@mariusandra
mariusandra deleted the new-job-queues branch May 26, 2021 15:38
}
const date = new Date(retry.timestamp).toISOString()
const [day, time] = date.split('T')
const dayTime = `${day.split('-').join('')}-${time.split(':').join('')}`

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

was there a specific reason for formatting things like this?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Basically just keeping it sortable to the second while removing special symbols that might interfere with it being a filename.

fuziontech pushed a commit to PostHog/posthog that referenced this pull request Oct 12, 2021
* store job queues with metadata

* basic S3 queue

* simplify job queues, create s3 wrapper

* test all parts of the s3 queue before connecting

* test s3 queue

* prettier

* fix timeout that ran just once
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

bump patch Bump patch version when this PR gets merged

Projects

None yet

Development

Successfully merging this pull request may close these issues.

S3 job queue

3 participants