Add pagination support and aggregate API results#138
Add pagination support and aggregate API results#138IvanildoBarauna wants to merge 2 commits intomainfrom
Conversation
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| def _extract_next_cursor(page: Any, next_cursor_key: str) -> Optional[str]: | ||
| """Extract the cursor value from a response payload.""" | ||
|
|
||
| if isinstance(page, dict): | ||
| cursor = page.get(next_cursor_key) | ||
| if isinstance(cursor, str) or cursor is None: | ||
| return cursor |
There was a problem hiding this comment.
Stop cursor pagination when cursor tokens are non‑strings
The new cursor_iterator only extracts the next cursor when the field value is a string. Many cursor-based APIs return integers or opaque objects. In _extract_next_cursor any non-string value is coerced to None, so cursor_iterator immediately breaks at the if next_cursor is None guard even though the response supplied a valid cursor. This causes multi-page APIs with numeric cursors to fetch just the first page and report pages == 1. Consider accepting any JSON-serialisable cursor instead of just strings so pagination can continue for endpoints that use integers.
Useful? React with 👍 / 👎.
Summary
Testing
poetry run pylint src/api_to_dataframepoetry run pytesthttps://chatgpt.com/codex/tasks/task_e_690ba1452b88832b9e0912eece6a9778