SimCLR.md

SimCLR

• Title: A Simple Framework for Contrastive Learning of Visual Representations
• Publication: ICML, 2020
• Link: [paper] [code]

Simple, but Outperform

Major Components

• composition of multiple data augmentations
• learnable nonlinear transformation
• contrastive cross entropy loss
• larger batch sizes and longer training

Contrastive Learning Framework

• maximize agreement between differently augmented views of the same data

1. Data augmentation : cropping, color distortions, Gaussian blur...
2. Neural network base encoder f(·) : extract representation vectors -> adopt ResNet
3. Small neural network projection head g(·) : use MLP with 1 hidden layer
4. Contrastive Loss function : Maximize agreement

Data Augmentation for Contrastive Representation Learning

Composition of data augmentation operations

• crucial for learning good representations
• spatial/geometric transformation : cropping, resizing, horizontal flipping, rotation, cutout
• appearance transformation : color distortion, color dropping, brightness, contrast, saturation, hue, Gaussian blur, Sobel filtering

Composing Augmentations

• no single transform suffices to learn good representations
• contrastive prediction task harder, but quality of representation improves dramatically
• stand out : random cropping & color distortion

Architectures for Encoder and Head

Unsupervised contrastive learning benefits (more) from bigger models

• unsupervised learning benefits more from bigger models than its supervised counterpart

A nonlinear projection head improves the representation quality of the layer before it

• using non-linear projection : better than linear projection & no projection

Loss Functions and Batch Size

Contrastive learning benefits (more) from larger batch sies and longer training

• larger batch sizes & training longer : provide more negative examples -> signifiant advantage

Reference

@article{DBLP:journals/corr/abs-2002-05709,
  author    = {Ting Chen and
               Simon Kornblith and
               Mohammad Norouzi and
               Geoffrey E. Hinton},
  title     = {A Simple Framework for Contrastive Learning of Visual Representations},
  journal   = {CoRR},
  volume    = {abs/2002.05709},
  year      = {2020},
  url       = {https://arxiv.org/abs/2002.05709},
  eprinttype = {arXiv},
  eprint    = {2002.05709},
  timestamp = {Fri, 14 Feb 2020 12:07:41 +0100},
  biburl    = {https://dblp.org/rec/journals/corr/abs-2002-05709.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}