Support for new/pretrained models

[moritzschaefer]

Report

This project/benchmark is awesome (I really enjoyed reading the paper) and I think it brings great value to the community.

I have two questions regarding the state of the repository:

1. "Benchmark your own model" listed as TODO in README: Can you provide an estimate, when the code base will be ready to be used for evaluation?
2. In line with point 1, the README provides a link to the an inference notebook image_model_loading.ipynb, but the notebook does not seem to provide information on how to load your evaluation data and assess benchmark results. Is there a way to run your benchmark on other/new models?

Versions

[rushin682]

Hi Moritz,

Glad you enjoyed the paper. To answer your Qs:

1. Rolling out new datasets and custom benchmarking on 31st August.
2. If I understand your question correctly, running the benchmark is not feasible using a jupyter notebook (i.e image_model_loading.ipynb). The jupyter notebook is provided to show how to load a model that has been pretrained using HESCAPE. This loaded model can be used for any downstream task you like for example, checkout HEST and HistoBistro.

Since the benchmark is massive, we have provided a hydra configuration script to keep track of all permutations of models. To make framework easier to use and read for new models, the tutorials will be updated in the next couple of weeks.

Thank you.

[moritzschaefer]

Great thank you for the info on your roadmap..that really helps

[rushin682]

Dear @moritzschaefer ,

The dataset has been updated on hugginface following the dataset usage policy. You can find the link for the data in README.md and here.

The custom benchmarking code is being worked on this week. Apologies for the delay.