Step-by-step pipeline for 3D VQA inference using checkpoints

[carottX]

Hi, thanks for the great work! I have downloaded the checkpoints and set up the environments for both repos. Could you clarify the exact pipeline to run 3D VQA (ScanQA/SQA3D) inference? A simple example command for a single scene would be very helpful. Thanks!

[ngailapdi]

Hi, thank you for your interest.

Please refer to this code base for instructions to run evaluation on downstream 3D VQA datasets.

[carottX]

Hi, thank you for your prompt reply!

I have a follow-up question regarding the inference pipeline:

Regarding language features:
Could you clarify how to obtain the language features used in decode_dataset.py? Specifically, are they generated from decode_features.py in the SplatTalk repo?

Regarding evaluation results:
I attempted an evaluation based on my own understanding of the pipeline, but the metrics do not seem to match those reported in the paper. Here are my results on ScanQA:

Metric	Score
EM1	0.0672
EM1 (refined)	0.2397
BLEU-1	0.1965
BLEU-2	0.0953
BLEU-3	0.0370
BLEU-4	~0.0000
METEOR	0.1300
ROUGE-L	0.2809
CIDEr	0.3661
SPICE	0.1578

I suspect the discrepancy may be due to an incorrect pipeline setup on my end. Any guidance on the correct end-to-end inference procedure would be greatly appreciated. Thanks!