Skip to content

Feat/theodw 2977 reduce memory usage#2439

Merged
HarrisonBoyleThomas merged 2 commits intomainfrom
feat/THEODW-2977-reduceMemoryUsage
Apr 13, 2026
Merged

Feat/theodw 2977 reduce memory usage#2439
HarrisonBoyleThomas merged 2 commits intomainfrom
feat/THEODW-2977-reduceMemoryUsage

Conversation

@HarrisonBoyleThomas
Copy link
Copy Markdown
Collaborator

@HarrisonBoyleThomas HarrisonBoyleThomas commented Apr 1, 2026

Ticket: https://pins-ds.atlassian.net/browse/THEODW-2977

  • Add SparkTestCase that has a function-level teardown to clear the pyspark cache
  • Ensure all tests that use pyspark inherit from the new test case
  • Update odw package pyspark version to 3.5 to match the spark3.5 pool
  • Update odw package delta-lake module to match spark3.5 pool

Copy link
Copy Markdown
Collaborator

@stef-solirius stef-solirius left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@HarrisonBoyleThomas HarrisonBoyleThomas force-pushed the feat/THEODW-2977-reduceMemoryUsage branch from 8c3fb0f to 2be485f Compare April 13, 2026 08:19
@HarrisonBoyleThomas HarrisonBoyleThomas merged commit e205774 into main Apr 13, 2026
9 checks passed
@HarrisonBoyleThomas HarrisonBoyleThomas deleted the feat/THEODW-2977-reduceMemoryUsage branch April 13, 2026 10:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants