Skip to content

Conversation

@ringoldsdev
Copy link
Owner

This pull request introduces significant enhancements to the Pipeline class in efemel/pipeline.py, focusing on improving chunked processing, adding type-safe overloads for flattening operations, and ensuring chunk size consistency throughout transformations. Additionally, it refactors import statements across multiple files for better readability and adds new configuration settings for the isort tool in pyproject.toml.

Enhancements to Pipeline functionality:

  • Chunk size consistency: Added chunk_size as a class attribute to ensure transformations like filter, map, tap_chunk, flatten, and chain preserve the original chunk size. ([[1]](https://github.com/ringoldsdev/efemel/pull/19/files#diff-41232c54a631af00f8efcf69ef8dd52bad12518bf2f443aaf36b3888182d22dcR58-R59), [[2]](https://github.com/ringoldsdev/efemel/pull/19/files#diff-41232c54a631af00f8efcf69ef8dd52bad12518bf2f443aaf36b3888182d22dcL65-R77), [[3]](https://github.com/ringoldsdev/efemel/pull/19/files#diff-41232c54a631af00f8efcf69ef8dd52bad12518bf2f443aaf36b3888182d22dcL114-R131), [[4]](https://github.com/ringoldsdev/efemel/pull/19/files#diff-41232c54a631af00f8efcf69ef8dd52bad12518bf2f443aaf36b3888182d22dcL259-R297), [[5]](https://github.com/ringoldsdev/efemel/pull/19/files#diff-41232c54a631af00f8efcf69ef8dd52bad12518bf2f443aaf36b3888182d22dcL277-R317))
  • Flatten method overloads: Introduced type-safe overloads for flatten to handle specific iterable types (list, tuple, set) while maintaining chunked structure for memory efficiency. ([efemel/pipeline.pyL165-R209](https://github.com/ringoldsdev/efemel/pull/19/files#diff-41232c54a631af00f8efcf69ef8dd52bad12518bf2f443aaf36b3888182d22dcL165-R209))
  • Extended flatten tests: Added new tests to verify chunked processing, chunk structure preservation, and chunk size maintenance during flattening operations. Removed redundant test for flattening string iterables. ([[1]](https://github.com/ringoldsdev/efemel/pull/19/files#diff-ef5f0938c9abd7d9ab658f28e363b093e59f677949829145bc7d6036cec3897fL405-L411), [[2]](https://github.com/ringoldsdev/efemel/pull/19/files#diff-ef5f0938c9abd7d9ab658f28e363b093e59f677949829145bc7d6036cec3897fR419-R484))

Refactoring and code cleanup:

  • Import statement reorganization: Refactored import statements across multiple files (efemel/cli.py, efemel/hooks_manager.py, efemel/process.py, efemel/readers/local.py, tests/test_cli.py) to enforce single-line imports and improve readability. ([[1]](https://github.com/ringoldsdev/efemel/pull/19/files#diff-de0f2600e6a42ac7b348702e57bd981acb52d415245acea28131a08442e9d630R1-R5), [[2]](https://github.com/ringoldsdev/efemel/pull/19/files#diff-31229c3b91b36bd395e96008601cbb24d075bf369ba4f374903be5a092e73291R2-L5), [[3]](https://github.com/ringoldsdev/efemel/pull/19/files#diff-f52c1cb1f15beaae0102d3ac011e78e66c2ed586365a8fb6365b81807ce45030L1-R4), [[4]](https://github.com/ringoldsdev/efemel/pull/19/files#diff-e795bca0bfc4ad93aa4345e6f014e2473469750c1f73785932c2bed526910d69L1-R2), [[5]](https://github.com/ringoldsdev/efemel/pull/19/files#diff-4e8715c7a425ee52e74b7df4d34efd32e8c92f3e60bd51bc2e1ad5943b82032eL2-R6))

Configuration updates:

  • isort settings: Added configuration in pyproject.toml to enforce single-line imports, order by type, and improve sorting rules for imports. ([pyproject.tomlR55-R63](https://github.com/ringoldsdev/efemel/pull/19/files#diff-50c86b7ed8ac2cf95bd48334961bf0530cdc77b5a56f852c5c61b89d735fd711R55-R63))

@ringoldsdev ringoldsdev merged commit d44b5a7 into main Jul 11, 2025
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants