Add support for GitHub pagination and ratelimiting#49
Merged
gene1wood merged 1 commit intomozilla:masterfrom Jan 3, 2019
Merged
Add support for GitHub pagination and ratelimiting#49gene1wood merged 1 commit intomozilla:masterfrom
gene1wood merged 1 commit intomozilla:masterfrom
Conversation
Create a new GitHubClient class which extends the Client class by modifying
the `request` method and creating four new methods
* Establish a new `paginate` parameter to the constructor
* Modify the `request` method to
* check if there aren't any allowed requests remaining due to ratelimiting
and if so wait the designated amount of time before making the request.
* if the `paginate` parameter is true, look in the payload of the response
page for a [link header](https://tools.ietf.org/html/rfc5988) and if found
extend the fetched data by calling the `get_additional_pages` method
* Create the `no_ratelimit_remaining` method to check if there are no
remaining requests allowed
* Create the `get_additional_pages` method which calls the
`get_next_link_url` method to determine if there are additional pages. If
so, fetch the next page and recursively continue fetching pages until the
last page is fetched
* Create the `get_next_link_url` method which parses the link header and
either returns the next page if there is one or an empty string if not
* Create the `ratelimit_seconds_remaining` method which returns the number
of seconds remaining until the ratelimit is cleared
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Create a new
GitHubClientclass which extends theClientclass by modifyingthe
requestmethod and creating four new methodspaginateparameter to the constructorrequestmethod toand if so wait the designated amount of time before making the request.
paginateparameter is true, look in the payload of the responsepage for a link header and if found
extend the fetched data by calling the
get_additional_pagesmethodno_ratelimit_remainingmethod to check if there are noremaining requests allowed
get_additional_pagesmethod which calls theget_next_link_urlmethod to determine if there are additional pages. Ifso, fetch the next page and recursively continue fetching pages until the
last page is fetched
get_next_link_urlmethod which parses the link header andeither returns the next page if there is one or an empty string if not
ratelimit_seconds_remainingmethod which returns the numberof seconds remaining until the ratelimit is cleared
This new functionality can be used like this
Based on conversations with jpaugh and nnja indicating we're going to redo the unit tests from scratch, I removed my in progress unit tests from this branch and figured I'd submit it now to make it available to folks despite the lack of unit tests.
This fixes #10