Conversation
DFDataset: do not pickle sharding info
|
I don't understand. We actually want that this is pickled/serialized. It doesn't make sense to not serialize this? We want/need this information. |
|
How does this work together with #1676? |
The dataset already handles the pickling of this information on its own via
#1676 removes the assertion, so the change here won't be necessary anymore. |
|
So instead of removing/cleaning up the obsolete/duplicate code/logic with |
|
If it unblocks @Judyxujj quickly then I see value in that, it's not like this PR has a huge number of lines :D |
|
Superseeded by #1676 |
... but rely on
_distrib_infoinstead.Fixes #1678
@Judyxujj feel free to give this PR a test run.