Make training take ~1 second

[andrew-healey]

Changes

Precomputing

The core training loop of PerSAM_f does a lot of unnecessary computation.
It evaluates the SAM mask detector once every epoch, but this step can be precomputed.
I modified the training loop to precompute this step.

Downsampling

Also, the current training loop uses mask images which are extremely high-resolution (some are 1280x1280 pixels).
I downsampled these to 256x256 pixels.

On Google Colab with a T4, these two changes decreased training time from ~9s to ~1s.

Caching SAM

Currently, the PerSAM_f script loads the full SAM model once for every image it trains on.
On Google Colab with a T4, this adds 8-15 seconds to the effective "training" time.
I modified the script to use a global SAM model, which it only loads once.

I have applied these same changes to persam_f_multi_obj.py. It seems to work, but I haven't put much focus on this, and I haven't evaluated its results.

Evaluation

Evaluation results are nearly identical on PerSeg.

Original:

Commit d4b42a8:
Args: Namespace(pred_path='persam_f', gt_path='./data/Annotations', ref_idx='00') 
backpack_dog, IoU: 95.76, Acc: 95.80
clock, IoU: 94.79, Acc: 94.84
colorful_teapot, IoU: 84.47, Acc: 84.67
dog3, IoU: 88.67, Acc: 93.88
duck_toy, IoU: 97.31, Acc: 97.31
fancy_boot, IoU: 95.96, Acc: 96.05
rc_car, IoU: 96.12, Acc: 96.25
round_bird, IoU: 96.85, Acc: 97.00
table, IoU: 94.66, Acc: 95.02
teddybear, IoU: 94.61, Acc: 96.23
wolf_plushie, IoU: 94.32, Acc: 95.74

Modified:

Commit f517cfd:
Args: Namespace(pred_path='persam_f', gt_path='./data/Annotations', ref_idx='00') 
backpack, IoU: 96.66, Acc: 96.77
backpack_dog, IoU: 95.76, Acc: 95.80
cat, IoU: 95.51, Acc: 95.87
cat2, IoU: 93.69, Acc: 93.77
cat_statue, IoU: 95.43, Acc: 96.17
chair, IoU: 92.14, Acc: 92.16
colorful_teapot, IoU: 84.84, Acc: 85.05
dog, IoU: 96.81, Acc: 96.87
dog4, IoU: 95.18, Acc: 95.43
dog6, IoU: 94.85, Acc: 95.00
dog7, IoU: 93.77, Acc: 94.11
duck_toy, IoU: 97.31, Acc: 97.31
fancy_boot, IoU: 95.96, Acc: 96.05
grey_sloth_plushie, IoU: 96.64, Acc: 96.71
monster_toy, IoU: 94.21, Acc: 94.28
table, IoU: 94.66, Acc: 95.02
teapot, IoU: 96.93, Acc: 97.19
teddybear, IoU: 94.61, Acc: 96.23
wolf_plushie, IoU: 94.32, Acc: 95.74

You can find the eval notebook in Colab here.

Note
The Colab environment runs OOM partway through both eval runs, causing some subjects in PerSeg not to be shown here.
This OOM error happens both before and after my proposed changes.