Optimise hdf5#36

Open

hllelli2 wants to merge 60 commits intomainfrom

Commits on Jul 2, 2024

feat: Add HDF5DataStore class for handling hdf5 files
hllelli2
committed

Commits on Jul 4, 2024

Add pytest configuration file and conftest.py for test fixtures
hllelli2
committed

Commits on Jul 9, 2024

Added a dataset config class to have all the defaults in one place
hllelli2
committed
Create a class for lazy-loading a hdf5 file
hllelli2
committed
Moved multi-processing to utils, Added an array dataset which can load arrays into the hdf5 without a file.
hllelli2
committed
Moved Multi-processing code here, code here to duplciate arrays from the dataloader, augment them and saves them to array. For a specific usecase in ml-toolkit
hllelli2
committed
Changed augment to take arrays instead of map-objects to make it compatible with ArrayDataset
hllelli2
committed
Removed mapObject transform base
hllelli2
committed
chore: Refactor transforms module and update transform classes
hllelli2
committed
first attempt at reducing if label/weight... is not none repeated code
hllelli2
committed
chore: Add test fixtures and update conftest.py for test setup
hllelli2
committed

Commits on Jul 10, 2024

set_gpu is not in the current ccpem-utils so check added JIC
hllelli2
committed

Commits on Oct 1, 2024

Commits on Oct 23, 2024

Commits on Oct 24, 2024

Cleanup: remove debug print statements from MapDataset class
hllelli2
committed

Commits on Dec 10, 2024

update pyproject.toml with dependency version
aj26git
committed

Commits on Mar 13, 2025

Enhance MapDataLoader: Add background filtering to skip slices with excessive background. Added check to see if train is set to false, if so, no augmentation is applied.
hllelli2
committed