Unable to join Discord + question about fine-tuning PaliGemma for fixed-prompt success detection

[YukiSato-cmd]

Hi, thank you for your great work on AutoEval.

I have two questions:

1. Discord invite
   I tried to join the Discord community, but the invite link does not seem to work for me.
   Could you let me know whether the Discord link is still valid, or if there is an updated invitation link?

2. Motivation for using PaliGemma for success detection
   After trying to implement the method myself, I had a question about the use of PaliGemma in AutoEval.

From my understanding, AutoEval uses PaliGemma for success detection by fine-tuning it with fixed prompts and fixed answer spaces, such as yes/no or sink/basket. In other words, it seems that the model is trained to respond to a fixed question format with a constrained set of outputs.

While implementing this, I started wondering about the motivation for using PaliGemma in this setting.

Since one of the strengths of a vision-language model is that it can handle diverse question forms and produce flexible outputs, I was curious why AutoEval uses it in a much more constrained way, where both the prompt format and the answer space are effectively fixed through fine-tuning.

In this setting, what would you say is the main advantage of using PaliGemma over a simpler task-specific classifier?
Is the main benefit flexibility across tasks, easier specification of success criteria in natural language, better robustness/generalization, or something else?

Thank you very much.

[zhouzypaul]

Hi,

Sorry about the Discord link, it may have expired. That channel is not actively maintained currently, feel free to ask questions here.

We use Paligemma basically as an initializtion for vision-language understanding, so we don't need to collect as many examples to train the classifier. It also should help with generalization from its internet scale pre-training. You can train a classifier from scratch to serve the same purpose, but you just may need many more training examples.

Hope that helps.

[YukiSato-cmd]

Thank you very much for your reply. That was very helpful.

I understand that the main advantage of using PaliGemma here is that it can reduce the amount of training data needed for success detection, and that its internet-scale pretraining may also provide better generalization. I also understand that a standard classifier trained from scratch could potentially achieve the same goal, but would likely require much more training data.

I have two additional questions regarding the setup.
From my understanding, AutoEval uses PaliGemma for success detection with fixed prompts and fixed answer spaces, such as yes/no or sink/basket. In other words, the model seems to be fine-tuned to answer a fixed question format with a constrained set of outputs.

First, is my understanding correct that both the prompt format and the output space are fixed in this setup?

Second, during inference, how sensitive is the method to changes in wording?
For example, if the original prompt is:
"is the drawer open? answer yes or no"
would the model still work properly if I changed it to a differently worded sentence with the same meaning, such as:
"Is the drawer currently open? Please answer yes or no."
or
"Would you say the drawer is open? Answer yes or no."
More generally, is the model robust to paraphrased prompts with the same meaning, or does it work best when the inference prompt closely matches the prompt format used during fine-tuning?

Thank you again for your time.

[zhouzypaul]

1. Yes, that's correct.
2. I haven't tried. It would work best when inference matches training since the model is finetuned. The models are public if you want to give it a try: https://huggingface.co/zhouzypaul/classifiers