Add S3 timeout parameters to ArrowCursor for role assumption support#610
Merged
laughingman7743 merged 2 commits intomasterfrom Oct 18, 2025
Merged
Conversation
This commit adds connect_timeout and request_timeout parameters to ArrowCursor and AsyncArrowCursor, addressing timeout issues when using role assumption with STS or experiencing high latency to S3. Changes: - Add connect_timeout and request_timeout parameters to ArrowCursor.__init__ - Add connect_timeout and request_timeout parameters to AsyncArrowCursor.__init__ - Pass timeout parameters to AthenaArrowResultSet - Update AthenaArrowResultSet to configure PyArrow S3FileSystem with timeout values - Add comprehensive docstrings explaining timeout parameters and use cases - Add tests for both int and float timeout values The default timeout values are None, which uses AWS SDK defaults (typically 1s for connect, 3s for request). Users experiencing timeout errors can now increase these values as needed. Fixes #609 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
- Update PyArrow minimum version from 7.0.0 to 10.0.0 (timeout parameters were added in Arrow 10.0.0 via ARROW-16521) - Add "S3 Timeout Configuration" section to Arrow documentation - Update README.rst with new PyArrow version requirement - Document timeout parameters usage examples for both ArrowCursor and AsyncArrowCursor 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
laughingman7743
added a commit
that referenced
this pull request
Oct 18, 2025
Update uv.lock to reflect the PyArrow version requirement change from >=7.0.0 to >=10.0.0 introduced in PR #610. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
connect_timeoutandrequest_timeoutparameters toArrowCursorandAsyncArrowCursorto resolve timeout issues when using role assumption with STS or experiencing high latency to S3.Problem
When using PyAthena's ArrowCursor with STS role assumption, S3 HeadObject operations can take longer than the default 3-second timeout, causing
NETWORK_CONNECTIONerrors (issue #609). This is particularly problematic when:Solution
This PR exposes PyArrow's S3FileSystem
connect_timeoutandrequest_timeoutparameters through PyAthena's ArrowCursor API, allowing users to configure appropriate timeout values for their environment.Changes
connect_timeoutandrequest_timeoutparametersconnect_timeoutandrequest_timeoutparametersUsage Example
Default Behavior
When timeout parameters are not specified (
None), PyArrow uses AWS SDK defaults:connect_timeout: ~1 secondrequest_timeout: ~3 secondsVersion Requirements
Important: This feature requires PyArrow >= 10.0.0, which added support for S3FileSystem timeout configuration (ARROW-16521). The minimum PyArrow version has been updated from 7.0.0 to 10.0.0 in:
pyproject.toml(botharrowextra anddevdependencies)README.rstTesting
Related Issues
Fixes #609
🤖 Generated with Claude Code