Skip to content

Conversation

@rguo123
Copy link
Collaborator

@rguo123 rguo123 commented Mar 13, 2025

use pyarrow filesystems for all file I/O to support s3 reads and writes

Copy link
Collaborator Author

rguo123 commented Mar 13, 2025

This stack of pull requests is managed by Graphite. Learn more about stacking.

@rguo123 rguo123 force-pushed the 03-13-handle_filesystems branch 4 times, most recently from f335a79 to 26d4fe9 Compare March 13, 2025 20:26
@rguo123 rguo123 marked this pull request as ready for review March 14, 2025 01:50
@rguo123 rguo123 requested a review from bmschmidt March 14, 2025 01:50
@rguo123 rguo123 marked this pull request as draft March 14, 2025 01:50
@rguo123 rguo123 removed the request for review from bmschmidt March 14, 2025 01:50
@rguo123 rguo123 force-pushed the 03-13-handle_filesystems branch 3 times, most recently from 34bb52b to b0f082d Compare March 19, 2025 19:57
@rguo123 rguo123 marked this pull request as ready for review March 19, 2025 19:58
@rguo123 rguo123 force-pushed the 03-13-handle_filesystems branch 2 times, most recently from c620b44 to a4077e8 Compare March 19, 2025 20:22
@bmschmidt bmschmidt force-pushed the 03-13-handle_filesystems branch from a4077e8 to db62935 Compare March 20, 2025 01:39
@rguo123 rguo123 force-pushed the 03-13-handle_filesystems branch 2 times, most recently from 6fc6bfd to 743dc47 Compare March 20, 2025 15:21
Copy link
Owner

@bmschmidt bmschmidt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, thanks

tile.overflow_loc.unlink()

# TODO: check with Ben that we should delete all the overflows here
for table in tables:
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this include the deletion of tile.overflow_loc unlink() too?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep! tables includes it

@rguo123 rguo123 force-pushed the 03-13-handle_filesystems branch from 743dc47 to 0cb6dbd Compare March 20, 2025 15:56
@rguo123 rguo123 merged commit b4ff60f into main Mar 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants