ML-helper-tools

This repo contains helper functions and skeletons for training ML models in an attempt to avoid repeating code search.

The idea is to have a single repo to refer to for implementation and editing instead of writting from scratch, or utillities that are not easily available.

This is not supposed to be a script repo.

Also installable as a pip module:

pip install git+ssh://git@github.com:AlexBarbera/ML-helper-tools.git  

Losses

Implementation of specific losses used in cool papers that have no direct implementation.

WassersteinLoss

• Earth-mover distance is a better metric for small permutations in image data compared to MSELoss.
• Original repo

PerceptualLoss

• Uses an encoder network to compare extracted features between target and generated, useful for SuperResolution. paper

TotalVariationLoss

• Jitters the image and calculates MSE.
• Used in SR paper

DeepEnergyLoss

• Maximizes difference between labels rather than set to 1 or 0.
• Docs

CycleGANLoss

• Extracted Cyclegan loss pipeline in case we want to abstract it to somthing different.

WordTreeLoss

• Based on YOLO9000 paper for hierarchical classification.
• Also utility class to use label hierarchy outside of training.

BinaryLabelSmoothing Wrapper

• Smoothes a one-hot vector by taking a bit of the top label and spreading it over the rest of the classes.
• Based on Improved Techniques for training GANs.
• Technically the smoothing would be $0 \rightarrow \frac{\epsilon}{k}$ and $1 \rightarrow \frac{k-1}{k}$
• While not a loss itself it is a loss wrapper.

 

Models

Standard models that generally are good to have in hand.

CycleGAN

• Unaligned datasets for domain tranformation.
• Original repo

Siamese Network

• Classifies differences in latent space of encoder.
• This is mine :)

ResNet Generator

• Standard ResNet
• Original repo

MultiLayer Perceptron (Linear and Convolutional)

• ¯\(ツ)/¯

 

Utils

A bunch of standalone functions that are commonly used.

LightningWrapper for training

• Allows to easily use distributed training and GPU/TPU usage.
• Pytorch Lightning

Scaling functions

• MinMax
• L2-norm
• Tensor to 8bit
• Z-score

TripletDataset

• Handles tuples of 3 (anchor [reference], positive [simmilar], negative [different]).

ABDataset

• Matches 2 classes of data in pairs.

MulticlassDataset

• Matches N classes of data in pairs.

ZCA Whitening

• Normalizes images so that covariance $\sum$ is the Identity matrix leading to decorrelated features.
• According to the paper, it should be applied batch-wise.

Adversarial Training

FGSM

• Fast Gradient Sign Method paper.
• Pytorch tutorial
• Uses the backward sign to generate noise to disrupt prediction.
• Generally 0.15 is a good delta.

I-FGSM

• Iterative FGSM paper.
• Iterates over deltas and predictions to greedely find min solution.

ILLCM

• Iterative Least-Likely Class Method paper
• Iterates the same way as I-FGSM but minimizing the least likely class.