Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 25 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,9 @@ The goal of this repo is to evaluate CLIP-like models on a standard set
of datasets on different tasks such as zero-shot classification and zero-shot
retrieval, and captioning.

Below we show the average rank (1 is the best, lower is better) of different CLIP models, evaluated
on different datasets.

![benchmark.png](benchmark.png)

The current detailed results of the benchmark can be seen [here](benchmark/README.md)
or directly in the [notebook](benchmark/results.ipynb).
- Results of OpenCLIP models on 38 datasets can be seen here: <https://github.com/mlfoundations/open_clip/blob/main/docs/openclip_results.csv>.
- Results from "Reproducible scaling laws for contrastive language-image learning" [[arXiv]](https://arxiv.org/abs/2212.07143) can be seen here: <https://github.com/LAION-AI/scaling-laws-openclip>
- Additional results are available in <https://github.com/LAION-AI/CLIP_benchmark_results>.

## Features

Expand All @@ -34,7 +30,7 @@ ViT-B-32,openai

to get the list of datasets
```
wget https://raw.githubusercontent.com/LAION-AI/CLIP_benchmark/main/benchmark/webdatasets.txt
wget https://raw.githubusercontent.com/LAION-AI/CLIP_benchmark/main/clip_benchmark/datasets/webdatasets.txt
```

Then to run
Expand Down Expand Up @@ -411,3 +407,24 @@ python setup.py install
- Thanks to [@teasgen](https://github.com/teasgen) for support of validation set and tuning linear probing similar to OpenAI's CLIP.
- Thanks to [@visheratin](https://github.com/visheratin) for multilingual retrieval datasets support from <https://arxiv.org/abs/2309.01859>.
- This package was created with [Cookiecutter](https://github.com/audreyr/cookiecutter) and the [audreyr/cookiecutter-pypackage](https://github.com/audreyr/cookiecutter-pypackage) project template. Thanks to the author.

## Citation

```
@software{cherti_2025_15403103,
author = {Cherti, Mehdi and
Beaumont, Romain},
title = {CLIP benchmark},
month = may,
year = 2025,
publisher = {Zenodo},
doi = {10.5281/zenodo.15403103},
url = {https://doi.org/10.5281/zenodo.15403103},
swhid = {swh:1:dir:8cf49a5dd06f59224844a1e767337a1d14ee56c2
;origin=https://doi.org/10.5281/zenodo.15403102;vi
sit=swh:1:snp:dd153b26f702d614346bf814f723d59fef3d
77a2;anchor=swh:1:rel:cff2aeb98f42583b44fdab5374e9
fa71793f2cff;path=CLIP\_benchmark-main
},
}
```
56 changes: 0 additions & 56 deletions benchmark/README.md

This file was deleted.

Loading