Skip to content

Fix numpy and pandas compatibility#475

Open
martin-springer wants to merge 59 commits intodevelopmentfrom
fix-numpy-and-pandas-compatibility
Open

Fix numpy and pandas compatibility#475
martin-springer wants to merge 59 commits intodevelopmentfrom
fix-numpy-and-pandas-compatibility

Conversation

@martin-springer
Copy link
Collaborator

@martin-springer martin-springer commented Feb 4, 2026

This pull request focuses on improving compatibility with recent versions of pandas (3.0+), numpy (2.x), and xgboost (3.x), while also removing deprecated normalization functions and addressing several minor bugs and deprecations. The main changes involve updating internal calculations to be resolution-agnostic, cleaning up deprecated code, and ensuring the codebase works with the latest dependencies.

Dependency and compatibility updates:

  • Removed upper version restrictions for pandas and numpy in setup.py to support pandas 3.0 and numpy 2.x; updated related requirements and changelog documentation.
  • Fixed pandas 3.0 compatibility throughout normalization.py by replacing .view('int64') with .total_seconds() for datetime calculations, making code resolution-agnostic and robust to changes in pandas datetime resolution. [1] [2] [3] [4] [5] [6]
  • Fixed numpy 2.x compatibility in soiling.py by using .item() and explicit indexing for scalar extraction, and replaced deprecated implicit conversions. [1] [2]

Deprecations and removals:

  • Removed deprecated normalization.delta_index and normalization.check_series_frequency functions (deprecated since v2.0.0); their private helpers remain for internal use. [1] [2] [3] [4]

Bug fixes and minor enhancements:

  • Ensured datetime resolution preservation in normalization.interpolate() so output matches input resolution (e.g., microseconds vs nanoseconds).
  • Fixed xgboost 3.x compatibility in filtering.xgboost_clip_filter() by using xgb.DMatrix with explicit feature names for model prediction. [1] [2]
  • Fixed pandas 4.0 deprecation warnings by switching Timedelta strings from lowercase 'd' to uppercase 'D' and using axis= keyword for DataFrame aggregation methods. [1] [2] [3] [4] [5]

Issues
Closes #472
Closes #445

These changes collectively ensure that the codebase remains compatible with the latest versions of key dependencies and removes legacy code, while also addressing minor bugs and improving code clarity.- [ ] Code changes are covered by tests

  • [ ] Code changes have been evaluated for compatibility/integration with TrendAnalysis
  • [ ] New functions added to __init__.py
  • API.rst is up to date, along with other sphinx docs pages
  • [ ] Example notebooks are rerun and differences in results scrutinized
  • Updated changelog

@martin-springer martin-springer changed the base branch from master to allow-clip-filter-for-energy February 4, 2026 15:36
@codecov-commenter
Copy link

codecov-commenter commented Feb 4, 2026

Codecov Report

❌ Patch coverage is 95.45455% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.14%. Comparing base (5c674a9) to head (fd434f9).

Files with missing lines Patch % Lines
rdtools/normalization.py 93.33% 2 Missing ⚠️
Additional details and impacted files
@@               Coverage Diff               @@
##           development     #475      +/-   ##
===============================================
- Coverage        96.22%   96.14%   -0.09%     
===============================================
  Files               12       12              
  Lines             2279     2280       +1     
===============================================
- Hits              2193     2192       -1     
- Misses              86       88       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@martin-springer martin-springer changed the title [WIP] Fix numpy and pandas compatibility Fix numpy and pandas compatibility Feb 4, 2026
@martin-springer
Copy link
Collaborator Author

martin-springer commented Feb 4, 2026

@mdeceglie - Commit 87d2b63 simplified the tz handling. Tests are already extensive with tz-naive and tz-aware unit tests. All are passing. The simplification gets us closer to the original implementation.

Base automatically changed from allow-clip-filter-for-energy to development February 4, 2026 22:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants