Skip to content

Add dataclass and pydantic support with simplified implementation#33

Merged
termoshtt merged 26 commits intomainfrom
dataclass-pydantic-support
Oct 25, 2025
Merged

Add dataclass and pydantic support with simplified implementation#33
termoshtt merged 26 commits intomainfrom
dataclass-pydantic-support

Conversation

@termoshtt
Copy link
Copy Markdown
Member

@termoshtt termoshtt commented Oct 25, 2025

This PR adds support for Python dataclasses and Pydantic models, building upon the work in #23 with significant architectural improvements.

Overview

This implementation extends deserialization capabilities to support:

  • Python dataclasses (standard library, Python 3.7+)
  • Pydantic models (popular data validation library)

Both types are now seamlessly converted to Rust structs via from_pyobject.

Key Differences from #23

Simplified Architecture

  • Removed caching mechanism: Eliminated once_cell dependency for simpler, more maintainable code
  • Separate modules: Split into dataclass.rs and pydantic.rs for better organization
  • Unified API pattern: Both modules follow the same *_as_dict() pattern returning Option<PyDict>

Improved Error Handling

  • Replaced .expect() calls with proper ? error propagation
  • Added graceful fallback when pydantic is not installed (returns None instead of panicking)
  • Added debug logging for better troubleshooting

Better Testing

  • Test matrix: CI now tests all combinations of pydantic presence and abi3 feature
  • Graceful test skipping: Pydantic tests automatically skip when pydantic is not installed
  • Comprehensive documentation: Added detailed module-level docs for each test suite

API Changes

  • Updated to PyO3 0.27.0 (from 0.26.0)
  • Replaced deprecated downcast() with cast() throughout
  • Replaced deprecated DowncastError with CastError

Implementation Details

Dataclass Support (src/dataclass.rs)

pub fn dataclass_as_dict<'py>(
    py: Python<'py>,
    obj: &Bound<'py, PyAny>,
) -> PyResult<Option<Bound<'py, PyDict>>>
  • Uses dataclasses.is_dataclass() to check if object is a dataclass
  • Converts to dict via dataclasses.asdict()
  • Returns None for non-dataclass objects

Pydantic Support (src/pydantic.rs)

pub fn pydantic_model_as_dict<'py>(
    py: Python<'py>,
    obj: &Bound<'py, PyAny>,
) -> PyResult<Option<Bound<'py, PyDict>>>
  • Attempts to import pydantic module (returns None if not available)
  • Checks if object is instance of BaseModel
  • Converts to dict via model_dump() method
  • Returns None for non-pydantic objects

Deserialization Flow (src/de.rs)

The deserializer now checks types in this order:

  1. Built-in types (dict, list, tuple, string, bool, int, float)
  2. Dataclass → convert via dataclass_as_dict()
  3. Pydantic model → convert via pydantic_model_as_dict()
  4. Objects with __dict__ attribute
  5. Objects with __slots__ attribute

Testing

New test files organized by Python object type:

  • tests/python_custom_class.rs - Custom classes with __dict__
  • tests/python_dataclass.rs - Dataclass tests (simple and nested)
  • tests/python_pydantic.rs - Pydantic model tests (auto-skips if not installed)

All tests follow the pattern:

  1. Define Python class/model
  2. Create instance
  3. Deserialize to Rust
  4. Compare with expected Rust value

Why Remove Caching?

The caching mechanism from #23 added complexity with minimal benefit:

  • Module imports are already cached by Python's import system
  • The performance gain was negligible for typical usage
  • Simpler code is easier to maintain and understand
  • Reduces dependencies (once_cell removed)

Migration from #23

If you were using the internal API from #23:

  • is_pydantic_base_model()pydantic_model_as_dict()
  • Return type changed from bool to Option<PyDict>
  • Module renamed: py_module_cachepydantic

The public API (from_pyobject, to_pyobject) remains unchanged.

Version Bump

Version bumped to 0.8.0 to reflect the new features and breaking changes in internal APIs.

LockedThread and others added 11 commits April 1, 2025 19:13
- Replace deprecated pyo3::PyObject with Py<PyAny>
- Add #[cfg(feature = "pydantic_support")] to is_module_installed to fix unused function warning
- Replace deprecated Python::with_gil with Python::attach in tests

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Fix typo: 'representtion' → 'representation' in src/de.rs
- Improve clarity of OnceCell safety comments in src/py_module_cache.rs

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Dataclasses are part of Python's standard library (Python 3.7+),
so there's no need to feature-gate them. This commit:

- Removes the `dataclass_support` feature flag
- Makes `once_cell` a regular dependency (always enabled)
- Removes all `#[cfg(feature = "dataclass_support")]` gates
- Keeps `pydantic_support` as an optional feature (external library)

Dataclass support is now always available by default.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@termoshtt termoshtt self-assigned this Oct 25, 2025
@termoshtt termoshtt changed the title Major Revise for #23 Pydantic/Dataclass support with major revise from #23 Oct 25, 2025
termoshtt and others added 14 commits October 25, 2025 15:47
This commit improves test organization and documentation:

**Test Documentation:**
- Add module-level documentation to all test files explaining their purpose
- Clarify test direction: check_revertible.rs tests Rust → Python → Rust only
- Clarify python_types.rs tests Python → Rust deserialization
- Document that reverse direction (Python → Rust → Python) is NOT tested

**Test Reorganization:**
- Split Python-specific type tests into separate file `tests/python_types.rs`
- Move 4 tests from `check_revertible.rs` to new file:
  * check_python_object (custom Python classes)
  * check_dataclass_object (dataclasses)
  * check_dataclass_object_nested (nested dataclasses)
  * check_pydantic_object (Pydantic models)

**Benefits:**
- Clearer separation of concerns between generic Rust types and Python types
- Explicit documentation of test coverage and limitations
- Easier to understand what each test suite covers
- Better maintainability as Python type support grows

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Split `tests/python_types.rs` into three specialized test files for better
organization and clarity:

**New Test Files:**
- `tests/python_custom_class.rs` - Custom Python classes with `__dict__`
  * Tests basic user-defined Python classes
  * 1 test: check_python_object

- `tests/python_dataclass.rs` - Python dataclasses (standard library)
  * Tests Python 3.7+ dataclasses
  * 2 tests: check_dataclass_object, check_dataclass_object_nested
  * Includes nested dataclass structures

- `tests/python_pydantic.rs` - Pydantic models (optional feature)
  * Tests Pydantic BaseModel subclasses
  * 1 test: check_pydantic_object
  * Requires `pydantic_support` feature
  * Uses `#![cfg(feature = "pydantic_support")]` at file level

**Changes:**
- Deleted `tests/python_types.rs`
- Added comprehensive documentation to each new test file explaining:
  * What Python type is being tested
  * How the type works in Python
  * The testing strategy used
- Updated references in `check_revertible.rs` to point to new files
- Applied feature gate at file level for pydantic tests

**Benefits:**
- Each Python type has its own dedicated test file
- Easier to locate and maintain tests for specific Python types
- Clear separation between standard library (dataclasses) and
  external library (Pydantic) features
- Better discoverability for contributors
- Pydantic tests are completely skipped when feature is disabled

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Configure CI to test all 4 combinations of pydantic (installed/not installed)
and abi3 (enabled/disabled) with fail-fast disabled to ensure comprehensive
coverage across different dependency configurations.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Rename py_module_cache.rs to pydantic.rs
- Remove caching mechanism (once_cell) for simpler implementation
- Implement pydantic_model_as_dict similar to dataclass_as_dict
- Replace expect() calls with proper error propagation
- Add debug logging when pydantic is not found
- Update de.rs to use unified pydantic_model_as_dict function

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
If pydantic module is not found, the test now returns early with
a skip message instead of failing. This allows tests to pass in
environments where pydantic is not installed.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@termoshtt termoshtt changed the title Pydantic/Dataclass support with major revise from #23 Add dataclass and pydantic support with simplified implementation Oct 25, 2025
@termoshtt termoshtt marked this pull request as ready for review October 25, 2025 09:27
Copilot AI review requested due to automatic review settings October 25, 2025 09:27
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds support for Python dataclasses and Pydantic models to the deserialization pipeline, enabling seamless conversion of these Python types to Rust structs. The implementation includes infrastructure improvements, better error handling, and comprehensive testing.

Key changes:

  • Added dataclass and Pydantic model deserialization support via new dataclass.rs and pydantic.rs modules
  • Updated to PyO3 0.27.0, replacing deprecated downcast() with cast() and DowncastError with CastError
  • Enhanced CI testing with matrix strategy covering pydantic presence and abi3 feature combinations

Reviewed Changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
src/dataclass.rs New module for dataclass-to-dict conversion using dataclasses.asdict()
src/pydantic.rs New module for Pydantic model-to-dict conversion using model_dump() with graceful fallback
src/de.rs Updated deserializer to check for dataclass/Pydantic types before __dict__ fallback; replaced deprecated downcast() with cast()
src/error.rs Replaced deprecated DowncastError with CastError
src/lib.rs Added module declarations for dataclass and pydantic
tests/python_dataclass.rs New test suite for dataclass deserialization (simple and nested)
tests/python_pydantic.rs New test suite for Pydantic model deserialization with auto-skip when unavailable
tests/python_custom_class.rs New test suite for custom Python classes, extracted from check_revertible.rs
tests/check_revertible.rs Added comprehensive documentation and removed custom class test (moved to dedicated file)
tests/to_json_to_pyobject.rs Added comprehensive documentation explaining cross-validation approach
Cargo.toml Version bump to 0.8.0, updated PyO3 to 0.27.0, added log dependency, removed abi3-py38 feature
.github/workflows/rust.yml Added test matrix for pydantic and abi3 combinations

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@termoshtt termoshtt merged commit ccad814 into main Oct 25, 2025
11 checks passed
@termoshtt termoshtt deleted the dataclass-pydantic-support branch October 25, 2025 09:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants