How to feed in image as inference input?

[Sapium59]

My understanding is that a single image, in form of output of inception-v3 encoder, can serve as input for inference input. I also noticed the pkl file of the pre-trained inception model you offered as a part of necessary files. I am newbie. I simply don't know how to utilize them. Is there some way to feed the image (or its latent code) into inference script?
Thank you!

[lalalune]

I am working on a solution for this. Currently inference is done by looking at the eval.txt in the training set, however this is hardcoded to the existing training sets. You can hack this by pointing to your own source image, but its not ideal.
I will add this functionality to this PR: #23

[SteveJunGao]

Hi @zhu-yuefeng,

We're doing unconditional generation, that means we only need to sample random Gaussian noise from the latent space, then our model is able to generate 3D shapes from it.

Therefore, we don't need to feed the image to inference.

The inception-v3 file is only used to compute FID metrics, it's not necessary if you only want to generate the 3D shapes.

Or maybe you can explain more details of what you want to try?

[Sapium59]

I am working on a solution for this. Currently inference is done by looking at the eval.txt in the training set, however this is hardcoded to the existing training sets. You can hack this by pointing to your own source image, but its not ideal. I will add this functionality to this PR: #23

Thank you for quick reply and great work! I can't wait to find this PR accepted.

[Sapium59]

Hi @SteveJunGao ,

Now I see you focus on unconditional generation. Thank you for to-the-point reply!

When I first read the paper, I misunderstood that "text-guided" generation with CLIP means "feed text into CLIP and use its output as z or w for generation". Now I think it means picking out one generated 3D model from many, while CLIP serves to 1) fine-tune a copy of your generator, and 2) score output models. Am I right this time?
One more question, will the text-guide module take considerable time, like in scale of unconditional generation itself?

As for my detailed motivate, for example, I have a photo of a car, and I want its textured 3D mesh. Random generation is pretty great, but I personally expect some "bias" rather than that.
I do not tend to make an image-guide module like you do with text. Maybe generating many candidate meshes and use inception-v3 to choose one would be enough, although I have not reached the inference step and thus no idea about time consumption. Any idea would help!

[SteveJunGao]

Re CLIP:
Yes, you're right, the CLIP model here only serves a loss function to fine-tune the copy of my 3D generator

Re time of text-guided module:
Yes, it will take much more time than unconditional generation as we're fine-tuning the model here.

Re generating a 3D mesh from the image:
This can be a great follow-up to our paper! Some paper you can look at is this one, although in our case, it might also need the camera that captures the image.

[Sapium59]

Thank you! @SteveJunGao

I close this issue, since I have got enough help and information about initial question.

I will keep you informed when anything occur.