Skip to content

Align PandasCursor chunk processing with PolarsCursor implementation #638

@laughingman7743

Description

@laughingman7743

Summary

Align PandasCursor's chunk processing implementation and documentation with the recently merged PolarsCursor implementation (#637).

Background

PolarsCursor was updated with improved chunk processing that:

  • Enables lazy loading for ALL data access methods (fetchone, fetchmany, fetchall) when chunksize is set
  • Provides iter_chunks() method on both Cursor and ResultSet
  • Has clear documentation explaining the lazy loading behavior

PandasCursor has the same underlying behavior but lacks:

  • Consistent API (iter_chunks() on ResultSet)
  • Clear documentation about lazy loading for fetch methods
  • as_pandas() method on DataFrameIterator

Tasks

Code Changes

  • Add iter_chunks() method to AthenaPandasResultSet
  • Add as_pandas() method to DataFrameIterator (align with Polars)
  • Update PandasCursor.iter_chunks() to delegate to ResultSet (like PolarsCursor)

Documentation Changes

  • Update docs/pandas.rst to clarify that chunksize enables lazy loading for all fetch methods
  • Add chunksize and iter_chunks() documentation to AsyncPandasCursor section

Import Fix

  • Change import abc to from collections import abc (same fix as PolarsCursor)

Related

🤖 Generated with Claude Code

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions