Some questions about the 'make_imnet_dataset' function

[LydJason]

Thank you very much for your great work!
I am trying to use the full dataset of IMnet and HSP to train ShapeFormer.
I discovered the make_imnet_dataset function in shapeformer/data/imnet_datasets/imnet_datasets.py write shape_vocab, vocab_idx, xbd and cat_id into the hdf5 file.
However, the code didn't write Ytg to the hdf5 file, which was provided in the processed dataset from google drive.
I was wondering maybe there are some other code that processes the original IMnet data to Ytg?
Thank you very much!

ShapeFormer/shapeformer/data/imnet_datasets/imnet_datasets.py

Line 350 in 3558e2a

dataDict = {"shape_vocab": np.array(shape_vocabs, dtype="O"), "vocab_idx": np.array(

[QhelDIV]

You can have a check at the Imnet2Dataset class at the file ShapeFormer/shapeformer/data/imnet_datasets/imnet_datasets.py. There are some snippets to decompress the shape_vocab back to Ytg.
Basically shape_vocab and vocab_idx is the HSP's representation to compress the Ytg volume (256x256x256).
The compression is actually really good and the storage is much smaller.

[LydJason]

I got it. Thank you very much!

[saladcat]

Thank you very much for your great work! I am trying to use the full dataset of IMnet and HSP to train ShapeFormer. I discovered the make_imnet_dataset function in shapeformer/data/imnet_datasets/imnet_datasets.py write shape_vocab, vocab_idx, xbd and cat_id into the hdf5 file. However, the code didn't write Ytg to the hdf5 file, which was provided in the processed dataset from google drive. I was wondering maybe there are some other code that processes the original IMnet data to Ytg? Thank you very much!

Thanks for your great works. Could you please provide the code for reducing the resolution from 256 to 64? I would like to keep my processing consistent with yours

[dqj5182]

@QhelDIV I am also interested in code for reducing the resolution of Ytg from 256 to 64.

[QhelDIV]

Hi @saladcat and @dqj5182 , thanks for the interest! I am not sure if I can find the original code for this.
But this should be simple as it is a mean-pooling process. You can use this (untested) code to do this with the einops package.

# Assuming Ytg has a shape of (256, 256, 256)
Ytg_x64 = einops.reduce(Ytg, '(x k1) (y k2) (z k3) -> x y z', 'mean', k1=8, k2=8, k3=8)

[dqj5182]

Thanks for the clarification!