Custom dataloader #162
Draft
Custom dataloader #162
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
e.g. resetting the state properly after ending a generator, saving state of sample buffer, skipping state of watchdog, log, gc dataset. Adapt required changes in tests. Add typing to specific saved states (instead of FlexState). Make debug output conditional.
Collaborator
philipp-fischer
left a comment
philipp-fischer
left a comment
There was a problem hiding this comment.
TODO: Check if we can add a check to detect users utilizing global RNG in taskencoder.
| def _pin_memory(self, sample: TSample) -> TSample: | ||
| return PinMemoryMixin.sample_pin_memory(sample, self._device) | ||
|
|
||
| def __call__(self, sample: Future[TSample]) -> Future[TSample]: |
Collaborator
There was a problem hiding this comment.
I believe this method is never called, because instead _pin_memory is directly being accessed. What's the idea behind the __call__ pattern?
Collaborator
Author
There was a problem hiding this comment.
See line 229 of dataloader.py:
self._pin_memory(
CallableFuture(functools.partial(self.restore_sample, sample_key))
)and line 317:
self._prefetching_samples[worker_idx].append(
self._pin_memory(worker.prefetch_next())
)… multiple shutdown/close functionalities, especially worker_init/_close
…custom_dataloader
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Implementing a custom dataloader instead of
torch.data.utils.DataLoader: