About VLMs

[YUECHE77]

Hi!

We are really interested in your work! While trying to reproduce the VLM results, I noticed that LLaVA-v1.5 uses openai/clip-vit-large-patch14-336 as its vision encoder, which is different from the LAION version.

Does this mean that, in order to reproduce LLaVA's results in Table 3, we also need to fine-tune openai/clip-vit-large-patch14-336? If so, could you please provide the fine-tuning script so that we can perfectly reproduce your results?

Meanwhile, could you please also upload the tar files for the other identities? For example, in Table 3: KANYE WEST, TOM CRUISE, BARACK OBAMA, LADY GAGA, and BRUCE LEE. Additionally, in Table 6 (Appendix): BASKETBALL, BEACH, CASTLE, REVOLVER, RIFLE, SCHOOLBUS, and SUNGLASSES. That would be very helpful!

Thank you so much!!

[YUECHE77]

In the meantime, regarding the Forget Accuracy evaluation in Table 3, the paper mentions:

"We sample 10 different targeted identities from the CelebA, with 100 images per identity, resulting in 1000 images in total, to create a comprehensive validation set."

I understand the process of sampling 10 different targeted identities from CelebA, which is straightforward. However, I assume you are not using the images from CelebA itself to construct the comprehensive validation set, since CelebA only contains about 10–20 images per person.

Could you please clarify which dataset you are sampling from, or could you upload the 1000 images used for this validation set?

I would really appreciate it if you could confirm this point. Thanks again!

[YtengTan]

Hi @YUECHE77,

Thanks for your interest in our work. Regarding your questions I have provided instructions/clarifications below.
Feel free to get back to me, if there's anything remain unclear to you.

VLM experiments

Our scrips for CLIP experiments are based on open_clip implementation. LLaVA-1.5 uses huggingface diffusers implementation, specifically, "openai/clip-vit-large-patch14-336" as its vision encoder.

1. To compute forget & retain gradients for huggingface models, run bash scripts/run_compute_grad_hf.sh (uncomment line-8 for LLaVA-1.5 CLIP).
2. To search pareto-front layers and step-sizes use SLUG/notebooks/experiment_vision_language_search.ipynb.
3. Our VLM evaluation is based on lmms-eval v0.3.0 code base. To evaluate unlearned VLMs, download and install lmms-eval following the instructions. Copy and paste SLUG/src/lmms-eval/lmms_eval/tasks folders to your downloaded lmms-eval/lmms_eval/tasks. Then, run

python3 -m accelerate.commands.launch \
    --num_processes=1 \
    -m lmms_eval \
    --model llava_hf \
    --model_args "ytan-ucr/llava-1.5-unlearned-elon-musk,device_map=cuda:0" \
    --tasks mu_elon \
    --batch_size 1 \
    --log_samples \
    --output_path ./logs/

you can pass --model_args "pretrained=llava-hf/llava-1.5-7b-hf,device_map=cuda:0" to evaluate original llava-1.5 accuracy on identifying Elon Musk images. For evaluation on other identity, pass --tasks mu_taylor,mu_mark etc., and pass --tasks mme,gqa,mmbench for VLM utility evaluation.

Note that due to the storage limitation, we have only provided the complete model weight for llava-1.5-unlearned-elon-musk on our huggingface. For other identities (e.g., Jeff Bezos, Tom Cruise), we have uploaded only the tuned layer weights: model-mu.safetensors and a corresponding model.safetensors.index.json. If you wish to run evaluation on those identities, you will need to download the complete llava-1.5 model directory, and move model-mu.safetensors and model.safetensors.index.json under each llava-1.5-unlearned-[celeb id] to the llava-1.5 model directory, so that evaluation script loads updated layer weights for unlearned identity.

Other tar files for supplementary experiments

We have uploaded other identity and object tar files here. If you wish to run on customized data set, you can use SLUG/src/clip/a0_create_tar.py to create your own tar files.

Clarification on VLM datasets

The 10 identities are sampled from CelebA label space based on their appearing frequency, this intuition is to avoid less frequent appeared celebrity that LLaVA/CLIP has poor classification performance on. We customize our own 1000 images data set on the selected 10-identity, on a basis that original LLaVA/CLIP have high accuracy on these data. The dataset is publicly available at https://huggingface.co/ytan-ucr/datasets.

[YUECHE77]

Hi @YtengTan ,

Thank you so much for your prompt reply and the detailed instructions!!! I really appreciate it.

However, I still have two follow up questions regarding the VLM experiments:

1. When training LLaVA-1.5's vision encoder (openai/clip-vit-large-patch14-336), why is the celebrity's name used as text input instead of the paired image description from the dataset? In other words, why are the training procedures different between the LAION version CLIP and LLaVA-1.5's vision encoder? Is there a specific reason for this inconsistency?

2. For the forget set when training the LAION version CLIP (script), the loss is:

(-1) * CosineEmbeddingLoss()(image_features, text_features, torch.ones(images.size(0)).to(device))

This makes sense, since we want to minimize the cosine similarity between image and text in the forget set. However, when training LLaVA-1.5's vision encoder (script), we don't have the negative sign. Could you clarify why the loss functions are different? Is this intentional, or did I miss something?

Looking forward to your clarification, and thanks again for your previous response!

[YUECHE77]

Hi @YtengTan ,

I followed the instruction but failed to reproduce your VLM results (the forget accuracy).

Could you please do a quick double check and run from your end to make sure all the VLM scripts are correct?

Thanks!