Cross dataset transferability

[alvitawa]

Hi, Thanks for your amazing work. Can I ask how to run the cross dataset transferability experiments with your repo?

Is this the correct command?

python train.py \
	--root ${DATA} \
	--seed ${SEED} \
	--trainer ${TRAINER} \
	--dataset-config-file configs/datasets/${DATASET}.yaml \
	--config-file configs/trainers/${CFG}.yaml \
	--output-dir ${DIR} \
	--model-dir ${MODEL_DIR} \
	--load-epoch ${LOADEP} \
	--eval-only \
	DATASET.NUM_SHOTS ${SHOTS} \
	DATASET.SUBSAMPLE_CLASSES ${SUB}

Or should I remove the last 2 rows like in https://github.com/muzairkhattak/multimodal-prompt-learning/blob/main/scripts/maple/xd_test_maple.sh

I couldnt find any information on how the tests are actually run.

[Rilliant7]

Hello, can you reproduce the performance claimed in the paper through his official code?

[PY-Lu]

@Rilliant7, It seems that I have encountered this issue as well. @alvitawa, Have you also faced this problem? How can we resolve it?

[alvitawa]

I ended up running it like this (See 4.):
https://github.com/alvitawa/fomo#training

You should be able to just copy those scripts into your repo and run them.

[Rilliant7]

@alvitawa Thank you for your response. Unfortunately, we have not been able to reproduce the performance mentioned in the original text. We have considered that the issue might be related to the GPU (4090 cuda12.2) and the conda version. Could you please let me know what GPU and CUDA version you are using? Additionally, would it be possible for you to provide a requirements.txt file for the environment you are using?

[alvitawa]

I havent reproduced their results on dataset transferability either (I havent tried). I just ran the dataset transfer experiment with my method, but the numbers I got seemed realistic. Do you have drastically different accuracies? Note also that the code I shared only uses seed=1.

I ran everything on A100 gpu, dont know the cuda version.

Requirements.txt is in the repo I shared.

[Rilliant7]

Thank you very much for your help. We will carefully investigate the reasons why we are unable to reproduce the results. Best wishes for you!

[Koorye]

@Rilliant7 @PY-Lu May I ask whether you have successfully reproduced the results on base-to-new task? My reproduced results are significantly different:

Base-to-New Generalization

Base	New	H
82.89	75.32	78.93

Offical report:

Base	New	H
84.00	77.23	80.48

I performed the above experiment on a V100 GPU, calculating the average of the 3 seeds.

[Rilliant7]

I haven't reproduced the performance claimed in the paper. The result of my reproduced is the same as yours: Base-to-New Generalization HM: 78.9. If you reproduce the official results later, I hope you can share them. Thank you.

…

---- Replied Message ---- | From | ***@***.***> | | Date | 06/19/2024 11:14 | | To | ShuvenduRoy/CoPrompt ***@***.***> | | Cc | Rilliant7 ***@***.***>, Mention ***@***.***> | | Subject | Re: [ShuvenduRoy/CoPrompt] Cross dataset transferability (Issue #3) | @***@***.*** May I ask whether you have successfully reproduced the results on base-to-new task? My reproduced results are significantly different: Base-to-New Generalization | Base | New | H | +-----+-----+--- | 82.89 | 75.32 | 78.93 | Offical report: | Base | New | H | +-----+-----+--- | 84.00 | 77.23 | 80.48 | — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[PY-Lu]

@Koorye, It appears that we have both encountered a similar issue, and I have also been able to reproduce a performance problem like yours. Therefore, could the author provide more details about the issue?

[Jacksong918]

Check if the learning rate is not consistent with the paper? LR=0.035

[zzeoZheng]

Check if the learning rate is not consistent with the paper? LR=0.035

But I think the LR=0.035 stated by the author in the paper is a typo, because this does not conform to the general setting, especially in the code which also states 0.0035