Few Shot Benchmark for Biomedical Datasets

Instruction of comet to run comet on Tabula Muris or Swissprot you will have to run the following commands:

python run.py exp.name=swissprot_pca_5shot method=comet dataset=swissprot_comet mode=train 

python run.py exp.name=tm_pca_5shot method=comet dataset=tabula_muris_comet mode=train 

in order to change how the concepts are create you will have to edit conf/dataset/swissprot_comet.yaml and conf/dataset/tabula_muris_comet.yaml by changing the following fields:

for swissprot

target: datasets.prot.concept_gen.LinkageConcepts

data_path: data/comet/swissprot/linkage.pt

for tabulas muris

target: datasets.cell.concept_gen.PCAConcepts

data_path: data/comet/tabula_muris/TM_PCA_components.pt

Installation

You have been provided with a fewshotbench.zip file containing the code for this benchmark. The accompanying presentation will also help you get started.

Conda

Create a conda env and install requirements with:

conda env create -f environment.yml 

Before each run, activate the environment with:

conda activate few-shot-benchmark 

Pip

Alternatively, for environments that do not support conda (e.g. Google Colab), install requirements with:

python -m pip install -r requirements.txt

Usage

Training

python run.py exp.name={exp_name} method=maml dataset=tabula_muris

By default, method is set to MAML, and dataset is set to Tabula Muris. The experiment name must always be specified.

Testing

The training process will automatically evaluate at the end. To only evaluate without running training, use the following:

python run.py exp.name={exp_name} method=maml dataset=tabula_muris mode=test

Run run.py with the same parameters as the training run, with mode=test and it will automatically use the best checkpoint (as measured by val ACC) from the most recent training run with that combination of exp.name/method/dataset/model. To choose a run conducted at a different time (i.e. not the latest), pass in the timestamp in the form checkpoint.time={yyyymmdd_hhmmss}. To choose a model from a specific epoch, use checkpoint.iter=40.

Datasets

We provide a set of datasets in datasets/. The data itself is not in the GitHub, but will either be automatically downloaded (Tabula Muris), or needs to be manually downloaded from here for the SwissProt dataset. These should be unzipped and put under data/{dataset_name}.

The configurations for each dataset are located at conf/dataset/{dataset_name}.yaml. To create a dataset, subclass the FewShotDataset class to create a SimpleDataset (for baseline / transfer-learning methods) and SetDataset (for the few-shot setting) and create a new config file for the dataset with the pointer to these classes.

The provided datasets are:

Dataset	Task	Modality	Type	Source
Tabula Muris	Cell-type prediction	Gene expression	Classification	Cao et al. (2021)
SwissProt	Protein function prediction	Protein sequence	Classification	Uniprot

Methods

We provide a set of methods in methods/, including a baseline method that does typical transfer learning, and meta-learning methods like Protoypical Networks (protonet), Matching Networks (matchingnet), and Model-Agnostic Meta-Learning (MAML). To create a new method, subclass the MetaTemplate class and create a new method config file at conf/method/{method_name}.yaml with the pointer to the new class.

The provided methods include:

Method	Source
Baseline, Baseline++	Chen et al. (2019)
ProtoNet	Snell et al. (2017)
MatchingNet	Vinyals et al. (2016)
MAML	Finn et al. (2017)

Models

We provide a set of backbone layers, blocks, and models in backbone.py, inclduing a 2-layer fully connected network as well as ConvNets and ResNets. The default backbone for each dataset is set in each dataset's config file, e.g. dataset/tabula_muris.yaml.

Configurations

This repository uses the Hydra framework for configuration management. The top-level configurations are specified in the conf/main.yaml file. Dataset-specific values are set in files in the conf/dataset/ directory, and few-shot method-specific files are specified in conf/method.

Note that the files in the dataset directory are at the top-level package, so configurations can be set at the command line directly, e.g. n_shot = 5 or backbone.layer_dim = [20,20]. However, configurations in conf/method are in the method package, which needs to be specified e.g. method.stop_epoch=20.

Note also that in Hydra, configurations are inherited through the specification of defaults. For instance, conf/method/maml.yaml inherits from conf/method/meta_base.yaml, which itself inherits from conf/method/method_base.yaml. Each configuration file then only needs to specify the deltas/differences to the file it is inheriting from.

For more on Hydra, see their tutorial. For an example of a benchmark that uses Hydra for configuration management, see BenchMD.

Experiment Tracking

We use Weights and Biases (WandB) for tracking experiments and results during training. All hydra configurations, as well as training loss, validation accuracy, and post-train eval results are logged. To disable WandB, use wandb.mode=disabled.

You must update the project and entity fields in conf/main.yaml to your own project and entity after creating one on WandB.

To log in to WandB, run wandb login and enter the API key provided on the website for your account.

References

Algorithm implementations based on COMET and CloserLookFewShot. Dataset preprocessing code is modified from each respective dataset paper, where applicable.

Slides and Additional Documentation

• How to integrate a dataset into the benchmark ?
• Slides (Available on Moodle)