how to use the image-guided trained weights?

[yanchuqiao]

Hi, Thank you for your greate wprk. I was able to train the model with image_guided and datatable input model. Now I am using this trained weights to test on a new dataset. but unlike the text_only models, this image-guided weigts only have .pth file, and doesn't include any other configration files.

I am not sure how to use this .pth file, where can I find other config files? It would be relaly helpful if you could provide some of instructions.

Thank you for your reply.

[aboggust]

Hi, thanks for your interest in VisText!

You can train and evaluate the image-guided models the same way you use the text-models.

For instance, the following command will run an image + scene graph model with prefix tuning:

bash run.sh -c image_guided -b 4 -e 50 -g 4 -i scenegraph -m byt5 -s 10 --prefix_tuning

Like you said, the pre trained weights for the image-guided models are on our dataset page. We also have the chart images and image features there in case you need it.

Let me know if there is anything else I can help with!

Angie