no 'tsdf' in npz file

[whitealex95]

Hi, when trying to run your code with 'TSDF' encoding, i get error on npz_file['tsdf']
Your npz file does not contain item of key 'tsdf', but it does have something on 'voxels'.
Are those the same?

[waterljwant]

Hi, try to load the npz file as follows.

with np.load(npz_file_path) as npz_file:
    rgb_tensor = npz_file['rgb']
    depth_tensor = npz_file['depth']
    tsdf_hr = npz_file['tsdf_hr'] 
    target_lr = npz_file['target_lr']

[whitealex95]

Hi, when I open you npz_file, I can only see *_voxels.npz, and when I open up one of them

npz_file = np.load('NYU0001_0000_voxels.npz')
npz_file.files

it only has
['position44', 'rgb_tesnor', 'target_lr', 'depth', 'voxels', 'position']
Am I looking at a wrong file?

[waterljwant]

Hi, when I open you npz_file, I can only see *_voxels.npz, and when I open up one of them

npz_file = np.load('NYU0001_0000_voxels.npz')
npz_file.files

it only has
['position44', 'rgb_tesnor', 'target_lr', 'depth', 'voxels', 'position']
Am I looking at a wrong file?

In that version, 'voxels' corresponds to flipped-TSDF encoded data.

[whitealex95]

Hi, thank you for the response!
I still have some question about getting the nonempty voxels with get_nonempty2 function

Here, does "nonempty" mean "occluded space + observed surface inside the view frustrum"?
Or, does it mean "All free space"
As far as I know in the Scene Completion task, I thought that the IoU does not take into account visible "Free space" nor voxels outside the frustrum

Also, could you tell me what the number 0.001 means when you are doing
data[voxels >= np.float32(0.001)] = 0 data[voxels ==1 ] = 1
in the get_empty function?

[waterljwant]

The "nonempty" voxels are the voxels in the occluded space and on the observed surface inside the view.

data[voxels ==1 ] = 1

The 'data' is the mask with binary value. The 1-valued elements represent the corresponding voxels are in "nonempty" space and 0-valued ones will be discarded for visualization.
During the evaluation, voxels in both "nonempty" space and free space will be taken into account.

[whitealex95]

Hi, thank you for the quick response.

1. If the voxels data is flipped-TSDF representation,
   how are the "Occuluded Region", "Free space" and "Out of the view frustrum" distinguished?

2. Also, in you npz_file['voxels'], which you said that it is a flipped-TSDF on the earlier response, there seems to be much more voxels with value 0.001 than voxels with value 1
   Is there a specific meaning on the value 0.001?

3. Could you also give me a hit on how the flipped tsdf representation are generated?
   For example, in NYUCADtrain_np/NYU0666_0000_voxels.npz,
   there are 5086259 voxels with value 0.001 while
   there are only 11895 voxels with value 1

[waterljwant]

The definition of flipped-TSDF can be found in paper SSCNet (https://sscnet.cs.princeton.edu/).
Our data were obtained from SSCNet. The value 0.001 is set according to the project as well.

[whitealex95]

Thank you for the response!
I managed to run your code, but there were several issues...

First, we were not able to reproduce the IoU in scene completion.
For completion task, we only got
precision:0.5102, recall: 1.0, IoU: 0.5102

I think the problem is on the target variable.
in your code

PALNet/SscDataLoader.py

Line 257 in f80bcaa

tsdf = self._downsample_tsdf(tsdf_hr, self.downsample)

you do downsampling of target_hr, which we obtained via .bin file.
We changed this part to use the downsampled target variable saved in the npz_file using:
target = npz_file['target_lr']
With this modification, we got a feasible IoU of 0.5102, but this is far behind the paper's value.
can you help us out with this problem?

The only other modification we made is on the npz loading part:
(

PALNet/SscDataLoader.py

Line 151 in f80bcaa

npz_file = np.load(_name + '.npz')

)

npz_file=np.load(_name + '_voxels.npz')
voxels = npz_file['voxels']
ftsdf=voxels
tsdf_hr = ftsdf.squeeze() 

[waterljwant]

We have trained the PALNet on the NYUCAD and then finetuned it on the NYU dataset.
Also, try to remove the voxels in the free space before calculating the IoU.