Implement better pagination for our GitHub Actions

[roryabraham]

If you haven’t already, check out our contributing guidelines for onboarding and email contributors@expensify.com to request to join our Slack channel!

---

Context

Our GitHub Actions interact with the GitHub API but lack pagination.

Action Performed:

There's not super clear reproduction steps here, but when generating the QA/deploy checklist (always labeled StagingDeployCash), we need to cross-check the list of PRs and make sure we don't include PRs created by our CI/CD bot user, OSBotify. We do that here.

However, we only fetch the latest 100 PRs, while there may be more than 100 PRs merged between deploy cycles (especially including all the automated bot ones). As a result, we include some automated PRs in the deploy checklist, which we do not want to do.

Expected Result:

We need to fetch all PRs between the oldest merged PR from the list passed in here and the newest PR. We can do that by fetching 100 PRs from the API at a time, until the oldest PR we have from the API is older than the oldest PR we were passed in.

Actual Result:

We don't fetch more than 100 results.

Workaround:

Ignore or manually delete automated PRs from the deploy checklist.

Platform:

Where is this issue occurring?

GitHub only.

view the job on upwork here.

View all open jobs on Upwork

[MelvinBot]

Triggered auto assignment to @jboniface (External), see https://stackoverflow.com/c/expensify/questions/8582 for more details.

[MelvinBot]

Triggered auto assignment to @johnmlee101 (Exported), see https://stackoverflow.com/c/expensify/questions/7972 for more details.

[FredericoGauz]

I will try to increase the number of pages retrieved by pagination or work on a recursive solution. I cannot guarantee this solution will work because it might be the case that Github prevents greater then 100 item numbers.

Here is an example with a different github api but similar case:
https://stackoverflow.com/questions/18148490/how-can-i-get-more-than-100-results-from-github-api-v3-using-github-api-gem

But I am happy to investigate and see what is possible to be done.

[roryabraham]

Yes, GitHub prevents greater than 100 items per fetch – you could use potentially use recursion to fetch all the results you need, but I would probably use a while loop.

[parasharrajat]

@roryabraham
Ok, If I understood it correctly, We already have a PRList which want to put on the body on the QA/deploy checklist but without the automated PRs.

So we call GitHub to get a list of all PRS and then Find AutomatedRequests out of those. Then we remove these found automated requests from the PRList. but the issue is We only get 100 requests.

So we want to get all PRs here so that we can filter all the Automated requests. Yeah, we can use https://octokit.github.io/rest.js/v18#pagination to get all the PRs which should solve the issue at hand.

But We will need to only get the PRs list that occurred between the oldest merged and Newest merged. SO we can quit early in https://octokit.github.io/rest.js/v18#pagination which allows us to stop at any time.
For Eg:

octokit.paginate("GET /organizations", (response, done) => {
  if (response.data.find((issues) => issue.body.includes("something"))) {
    done();
  }
  return response.data;
});

[roryabraham]

Nice find with https://octokit.github.io/rest.js/v18#pagination! Ideally we could make the paginator fetch all the results we need and return a single promise, such that we:

1. Fetch all the PRs at until we get to PRs older than the oldest one on our PRList
2. Then remove any automated ones from our PRList
3. Then continue and generate the StagingDeployCash issue body.

[roryabraham]

Maybe you could use promiseWhile to create a wrapper around the paginated method or something

[parasharrajat]

octokit.github.io/rest.js/v18#pagination! already return a promise So I don't think we need that.