This repository provides source code for the COMPAYL25 workshop paper "Effortless VLM Specialization in Histopathology without Annotation" [paper
] [blogpost].
Abstract: Recent advances in Vision-Language Models (VLMs) in histopathology, such as CONCH and QuiltNet, have demonstrated impressive zero-shot classification capabilities across various tasks. However, their general-purpose design may lead to suboptimal performance in specific downstream applications. To address this limitation, several supervised fine-tuning methods have been proposed, which require manually labeled samples for model adaptation. This paper investigates annotation-free adaptation of VLMs through continued pretraining on domain- and task-relevant image-caption pairs extracted from existing databases. Our experiments on two VLMs, QuiltNet and CONCH, across three downstream tasks reveal that these pairs significantly enhance both zero-shot and few-shot performance. Notably, continued pretraining achieves comparable few-shot performance with larger training sizes, leveraging its task-agnostic and annotation-free nature to facilitate model adaptation for new tasks.
This repository is built upon the [CONCH repository], please follow the installation guidelines there and install additional packages in the requirements.text.
- Retrieve relevant domain/task-specific image-caption pairs:
python code/retrieve.py- Domain/Task-specific continued pretraining:
python code/finetune.py- Few-shot learning (CoOp):
python code/coop.py