Implicit bias produces neural scaling laws in learning curves, from perceptrons to deep networks

Requirements

To install requirements:

conda env create --file deep_norm-environment.yml
conda activate deep_norm

To download datasets: python download_datasets.pbs

Training

To train a single run of a model and dataset used in the paper, run this command:

python CNN_train.py --P 48000 --T 500 --lr 1e-3 --seed 11130 --dataset CIFAR10 --data_root ./data --out_dir <select_directory> 

To obtain the curves in the paper it is necessary to run for different P values and seed choices.

Results presented in the paper

All data necessary to reproduces curves of experiments in deep networks are reproducible via the aggregated curves over many seeds in folder ./analysis. Inside folder ./analysis, plots in the paper are reported in the notebook Graphs_deep_networks_experiments.ipynb, while all intermediate analysis to aggregate data and produce curves are reported in notebooks Main_analysis_notebook.ipynb and Analysis_WD_SGD_NORMS.ipynb