Skip to content

Optimise hdf5 #36

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
hllelli2 wants to merge 60 commits into
base: main
Choose a base branch
from
Open

Optimise hdf5 #36

hllelli2 wants to merge 60 commits into from

Conversation

@hllelli2
Copy link
Collaborator

@hllelli2 hllelli2 commented Oct 1, 2024

  • MapDataset, ArrayDataset and MapDataloader to handle inputs of .mrc or .map data.
  • Transformations and Augmentations build upon CCPEM_Utils.
  • Hdf5 intermediate data store
  • Added accompanying tests
  • Added basic pip install

hllelli2 added 30 commits June 19, 2024 16:08
@hllelli2
Updates to gitignore
c64fbff
@hllelli2
basic setup config for pip install
2f4d58f
@hllelli2
Added MapDataset and MapDataLoader
6afd99e
@hllelli2
Added custom augments and transfroms as well as tiling
ef73dd2
@hllelli2
Added tests for new code
2ab4260
@hllelli2
feat: Add HDF5DataStore class for handling hdf5 files
8b9d06d
@hllelli2
Add pytest configuration file and conftest.py for test fixtures
d8729a1
@hllelli2
Added a dataset config class to have all the defaults in one place
1a6ed24
@hllelli2
Create a class for lazy-loading a hdf5 file
c8486b8
@hllelli2
Moved multi-processing to utils, Added an array dataset which can loa…
3779ee4 
…d arrays into the hdf5 without a file.
@hllelli2
Moved Multi-processing code here, code here to duplciate arrays from …
b9a91cb 
…the dataloader, augment them and saves them to array. For a specific usecase in ml-toolkit
@hllelli2
Changed augment to take arrays instead of map-objects to make it comp…
ab2ce2f 
…atible with ArrayDataset
@hllelli2
Removed mapObject transform base
d6eda3a
@hllelli2
chore: Refactor transforms module and update transform classes
a86c910
@hllelli2
first attempt at reducing if label/weight... is not none repeated code
41d40b3
@hllelli2
chore: Add test fixtures and update conftest.py for test setup
6f73aae
@hllelli2
set_gpu is not in the current ccpem-utils so check added JIC
07e2318
@hllelli2
chore: Refactor HDF5DataStore class and add support for temporary dir…
a6cc9b1 
…ectory storage
@hllelli2
Removed "_" id logic in MapDataLoader, might reimplement later but no…
05f792b 
…t sure if needed for multi-class porblems
@hllelli2
Refactor process_datasets function and remove unnecessary code
5a770f7
@hllelli2
Refactor transforms module and update transform classes
0a99fd7
@hllelli2
re-added __del__ method to HDF5Store
8772fa6
@hllelli2
Refactor MapDataset and ArrayDataset classes to handle weight tensors
93c8424
@hllelli2
Refactor
9128e9a
@hllelli2
Refactor DecomposeToSlices, MapObjectMaskCrop, and MapObjectPadding c…
b2a49c1 
…lasses
@hllelli2
Refactor test_map_io.py to handle weight tensors and update dataset l…
ce47030 
…engths
@hllelli2
Refactor HDF5DataStore class to handle weight tensors and implement c…
5d34da0 
…aching mechanism
@hllelli2
pre-commit fixes
d049499
@hllelli2
Refactor MapDataLoader to add caching mechanism for HDF5DataStore
1e65dd8
@hllelli2
Refactor HDF5DataStore class to handle weight tensors and update cach…
f31870b 
…ing mechanism
@hllelli2 hllelli2 self-assigned this Oct 1, 2024
aj26git
aj26git reviewed Oct 2, 2024
View reviewed changes
aj26git
aj26git reviewed Oct 3, 2024
View reviewed changes
aj26git
aj26git reviewed Oct 3, 2024
View reviewed changes
aj26git
aj26git reviewed Oct 3, 2024
View reviewed changes
aj26git
aj26git reviewed Oct 3, 2024
View reviewed changes
@aj26git
Copy link
Collaborator

aj26git commented Oct 3, 2024

@hllelli2 : Overall it looks good. I have added a few comments/suggestions. It is good to go once these are addressed.

hllelli2 and others added 12 commits October 9, 2024 13:53
@hllelli2
Add iterator method to HDF5DataStore and simplify key checks in tests
0696806
@hllelli2
Refactor test fixtures to use default scope and improve readability
7fbefdf
@hllelli2
Refactor: Rename tiles to slice_indicies and update related logic
36a5883
@hllelli2
Refactor: Simplify iterator method in HDF5DataStore
b24d299
@hllelli2
Refactor: update default voxel parameter type and improve kwargs hand…
7d12ece 
…ling
@hllelli2
Merge branch 'OptimiseHdf5' of https://github.com/alan-turing-institu…
f4555ae 
…te/CAKED into OptimiseHdf5
@hllelli2
Updated MapDataLoader to accepted the Mapdataset Kwargs in the load f…
2f32536 
…unction. This allows any keywords to be applied to all MapDatasets.
@hllelli2
Fix: Change default value of 'vox' to a float and update related logi…
1844ec7 
…c in MapObjectVoxelNormalisation
@hllelli2
Merge branch 'OptimiseHdf5' of https://github.com/alan-turing-institu…
96772ae 
…te/caked into OptimiseHdf5
@hllelli2
Cleanup: remove debug print statements from MapDataset class
b8d9c65
@aj26git
update pyproject.toml with dependency version
d975d44
@hllelli2
Enhance MapDataLoader: Add background filtering to skip slices with e…
dc23c5c 
…xcessive background. Added check to see if train is set to false, if so, no augmentation is applied.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@aj26git aj26git aj26git left review comments

Assignees

@hllelli2 hllelli2

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@hllelli2 @aj26git