Skip to content

Conversation

@Christos-Hadjinikolis
Copy link
Collaborator

@Christos-Hadjinikolis Christos-Hadjinikolis commented Apr 23, 2025

Summary

This PR refactors the WithAthena mixin to use [awswrangler.athena.read_sql_query](https://aws-sdk-pandas.readthedocs.io/en/stable/stubs/awswrangler.athena.read_sql_query.html) for all Athena reads, replacing the previous PyAthena-based implementation.


✅ What's changed

  • 🔄 Replaced PyAthena with AWS Wrangler in WithAthena
  • ✅ Refactored internal _run_query_run_wr_athena_query_wrapped
  • 🎯 Introduced argument filtering via @allow_options directly on the Wrangler reader
  • 📦 Updated AthenaDataSubSection schema:
    • s3_staging_dirs3_output
    • Removed region_name since it's optional for Wrangler
  • 🧪 Updated tests:
    • Switched from mocking pyathena.connect to wr.athena.read_sql_query
    • Asserted argument propagation and output equivalence
  • 🧼 Removed unused or legacy imports related to PyAthena
  • 🔄 Updated all YAML resource definitions referencing Athena
  • 🔁 Applied consistent allow_options(...) usage across mixins

💡 Why this matters

  • Removes dependency on PyAthena
  • Aligns Athena I/O with modern Wrangler standards used in other mixins
  • Enables better support for caching, CTAS, UNLOAD, and batching
  • Makes argument filtering consistent and future-proof
  • Reduces maintenance overhead and surface area of custom logic

⚠️ Notes

  • Writing to Athena is still unsupported, as expected;
  • region and boto3_session are now not mandatory fields;
  • s3_additional_kwargs and pyarrow_additional_kwargs are handled properly via @allow_options.

@sonarqubecloud
Copy link

@codecov
Copy link

codecov bot commented Apr 23, 2025

Codecov Report

Attention: Patch coverage is 97.72727% with 1 line in your changes missing coverage. Please review.

Project coverage is 89.31%. Comparing base (9cb7282) to head (a755044).

Files with missing lines Patch % Lines
dynamicio/mixins/with_athena.py 91.66% 1 Missing ⚠️
Additional details and impacted files
@@              Coverage Diff              @@
##           rnd-13624      #89      +/-   ##
=============================================
- Coverage      89.40%   89.31%   -0.09%     
=============================================
  Files             16       16              
  Lines           1321     1310      -11     
  Branches         153      154       +1     
=============================================
- Hits            1181     1170      -11     
  Misses            98       98              
  Partials          42       42              
Flag Coverage Δ
unittests 89.31% <97.72%> (-0.09%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@Christos-Hadjinikolis Christos-Hadjinikolis changed the title refactor: Migrate WithAthena mixin to aws-wrangler refactor: Migrate WithAthena mixin to use AWS Wrangler for Athena I/O Apr 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant