not convergence

[robotzheng]

./examples/train_fp16.sh aognet_s configs/aognet_imagenet_12M.yaml first_try

we use 4 P40 GPUs, so change the "train_fp16.sh" like this:

Change accordingly

GPUS=0,1,2,3
NUM_GPUS=4
NUM_WORKERS=4

the log is below:
FP16_Optimizer received torch.cuda.FloatTensor with torch.Size([206])
FP16_Optimizer received torch.cuda.HalfTensor with torch.Size([824, 206, 1, 1])
FP16_Optimizer received torch.cuda.FloatTensor with torch.Size([824])
FP16_Optimizer received torch.cuda.FloatTensor with torch.Size([824])
FP16_Optimizer received torch.cuda.FloatTensor with torch.Size([824])
FP16_Optimizer received torch.cuda.FloatTensor with torch.Size([824])
FP16_Optimizer received torch.cuda.FloatTensor with torch.Size([824])
FP16_Optimizer received torch.cuda.HalfTensor with torch.Size([1000, 824])
FP16_Optimizer received torch.cuda.FloatTensor with torch.Size([824])
FP16_Optimizer received torch.cuda.FloatTensor with torch.Size([824])
FP16_Optimizer received torch.cuda.FloatTensor with torch.Size([824])
FP16_Optimizer received torch.cuda.HalfTensor with torch.Size([1000, 824])
FP16_Optimizer received torch.cuda.HalfTensor with torch.Size([1000])
FP16_Optimizer received torch.cuda.HalfTensor with torch.Size([1000])
torch.Size([256, 3, 224, 224])
torch.Size([256, 3, 224, 224])
torch.Size([256, 3, 224, 224])
torch.Size([256, 3, 224, 224])
Epoch: [0][0/1252] Time 12.503 (12.503) Speed 81.903 (81.903) Data 0.984 (0.984) Loss 6.9218750000 (6.9219) Prec@1 0.195 (0.195) Prec@5 0.586 (0.586) lr 0.000064
Epoch: [0][100/1252] Time 2.100 (2.206) Speed 487.515 (464.191) Data 0.001 (0.011) Loss nan (nan) Prec@1 0.000 (0.003) Prec@5 0.000 (0.009) lr 0.006454
Epoch: [0][200/1252] Time 2.116 (2.155) Speed 483.939 (475.125) Data 0.001 (0.006) Loss nan (nan) Prec@1 0.000 (0.001) Prec@5 0.000 (0.004) lr 0.012843
Epoch: [0][300/1252] Time 2.103 (2.137) Speed 487.036 (479.160) Data 0.001 (0.004) Loss nan (nan) Prec@1 0.000 (0.001) Prec@5 0.000 (0.003) lr 0.019233

[robotzheng]

close fp16 also not convergence.

the log is below:
Epoch: [27][1100/2503] Time 1.158 (1.165) Speed 442.331 (439.497) Data 0.001 (0.001) Loss 6.9046459198 (6.9073) Prec@1 0.000 (0.097) Prec@5 0.586 (0.490) lr 0.181792
Epoch: [27][1200/2503] Time 1.156 (1.164) Speed 442.782 (439.748) Data 0.001 (0.001) Loss 6.9092612267 (6.9072) Prec@1 0.000 (0.097) Prec@5 0.195 (0.491) lr 0.181729
Epoch: [27][1300/2503] Time 1.174 (1.165) Speed 436.201 (439.646) Data 0.001 (0.001) Loss 6.9077682495 (6.9073) Prec@1 0.000 (0.098) Prec@5 0.195 (0.490) lr 0.181666
Epoch: [27][1400/2503] Time 1.175 (1.165) Speed 435.583 (439.619) Data 0.001 (0.001) Loss 6.9074120522 (6.9072) Prec@1 0.000 (0.098) Prec@5 0.391 (0.492) lr 0.181603
Epoch: [27][1500/2503] Time 1.156 (1.165) Speed 442.869 (439.624) Data 0.000 (0.001) Loss 6.9094963074 (6.9072) Prec@1 0.000 (0.099) Prec@5 0.000 (0.491) lr 0.181540
Epoch: [27][1600/2503] Time 1.156 (1.165) Speed 442.899 (439.625) Data 0.001 (0.001) Loss 6.9062004089 (6.9072) Prec@1 0.391 (0.099) Prec@5 1.562 (0.495) lr 0.181476
Epoch: [27][1700/2503] Time 1.175 (1.164) Speed 435.766 (439.706) Data 0.001 (0.001) Loss 6.9037847519 (6.9072) Prec@1 0.000 (0.098) Prec@5 0.586 (0.494) lr 0.181413
Epoch: [27][1800/2503] Time 1.156 (1.165) Speed 442.896 (439.624) Data 0.001 (0.001) Loss 6.9067974091 (6.9072) Prec@1 0.195 (0.099) Prec@5 0.391 (0.496) lr 0.181350
Epoch: [27][1900/2503] Time 1.156 (1.164) Speed 442.955 (439.792) Data 0.000 (0.001) Loss 6.9101543427 (6.9072) Prec@1 0.195 (0.099) Prec@5 0.391 (0.497) lr 0.181286
Epoch: [27][2000/2503] Time 1.156 (1.164) Speed 442.931 (439.942) Data 0.001 (0.001) Loss 6.9117798805 (6.9072) Prec@1 0.391 (0.099) Prec@5 0.391 (0.497) lr 0.181223

[tfwu]

Please use our new repo: https://github.com/iVMCL/AOGNet-v2
Please let us know if you still have problems.