My training vs Paper training

[brumocas]

After playing around with DOPE I decided to train the cracker 3D object to compare my training with the weights obtained from the training for the project paper.

3D Object

Dataset
I generated 83K images with blenderprocfollowing the guidelines provided in this repo/issues.

Training
I have been training this object for a couple days and my loss looks like this:

Train Epoch: 29 [35000/83967 (42%)]     Loss: 0.008937404491007
Train Epoch: 29 [40000/83967 (48%)]     Loss: 0.007676803041250
Train Epoch: 29 [45000/83967 (54%)]     Loss: 0.007751216180623
Train Epoch: 29 [50000/83967 (60%)]     Loss: 0.007393791805953
Train Epoch: 29 [55000/83967 (65%)]     Loss: 0.007743947207928
Train Epoch: 29 [60000/83967 (71%)]     Loss: 0.007903006859124
Train Epoch: 29 [65000/83967 (77%)]     Loss: 0.008371086791158
Train Epoch: 29 [70000/83967 (83%)]     Loss: 0.008013324812055
Train Epoch: 29 [75000/83967 (89%)]     Loss: 0.007521728985012
Train Epoch: 29 [80000/83967 (95%)]     Loss: 0.007015910465270
Train Epoch: 30 [0/83967 (0%)]  Loss: 0.007150554563850
Train Epoch: 30 [5000/83967 (6%)]       Loss: 0.007973545230925
Train Epoch: 30 [10000/83967 (12%)]     Loss: 0.007326924707741
Train Epoch: 30 [15000/83967 (18%)]     Loss: 0.007029242813587
Train Epoch: 30 [20000/83967 (24%)]     Loss: 0.007687418721616
Train Epoch: 30 [25000/83967 (30%)]     Loss: 0.007528588175774
Train Epoch: 30 [30000/83967 (36%)]     Loss: 0.008784147910774
Train Epoch: 30 [35000/83967 (42%)]     Loss: 0.008270633406937
Train Epoch: 30 [40000/83967 (48%)]     Loss: 0.008088813163340
Train Epoch: 30 [45000/83967 (54%)]     Loss: 0.007582063321024
Train Epoch: 30 [50000/83967 (60%)]     Loss: 0.007671120576560
Train Epoch: 30 [55000/83967 (65%)]     Loss: 0.006739565171301
Train Epoch: 30 [60000/83967 (71%)]     Loss: 0.008357232436538
Train Epoch: 30 [65000/83967 (77%)]     Loss: 0.007158810272813
Train Epoch: 30 [70000/83967 (83%)]     Loss: 0.009133071638644
Train Epoch: 30 [75000/83967 (89%)]     Loss: 0.009372244589031
Train Epoch: 30 [80000/83967 (95%)]     Loss: 0.008244232274592
WARNING:tensorboardX.x2num:NaN or Inf found in input tensor.
Train Epoch: 31 [0/83967 (0%)]  Loss: 0.008372079581022
Train Epoch: 31 [5000/83967 (6%)]       Loss: 0.008861823007464
Train Epoch: 31 [10000/83967 (12%)]     Loss: 0.009239616803825
Train Epoch: 31 [15000/83967 (18%)]     Loss: 0.008081949315965
Train Epoch: 31 [20000/83967 (24%)]     Loss: 0.007703350391239
Train Epoch: 31 [25000/83967 (30%)]     Loss: 0.008221860043705
Train Epoch: 31 [30000/83967 (36%)]     Loss: 0.008797293528914
Train Epoch: 31 [35000/83967 (42%)]     Loss: 0.007885038852692
Train Epoch: 31 [40000/83967 (48%)]     Loss: 0.006942533887923
Train Epoch: 31 [45000/83967 (54%)]     Loss: 0.007813187316060
Train Epoch: 31 [50000/83967 (60%)]     Loss: 0.008053245022893
Train Epoch: 31 [55000/83967 (65%)]     Loss: 0.008084481582046
Train Epoch: 31 [60000/83967 (71%)]     Loss: 0.008321086876094
Train Epoch: 31 [65000/83967 (77%)]     Loss: 0.007608712650836
Train Epoch: 31 [70000/83967 (83%)]     Loss: 0.008138785138726
Train Epoch: 31 [75000/83967 (89%)]     Loss: 0.005781981162727
Train Epoch: 31 [80000/83967 (95%)]     Loss: 0.009008388966322
Train Epoch: 32 [0/83967 (0%)]  Loss: 0.007689560763538
Train Epoch: 32 [5000/83967 (6%)]       Loss: 0.008272772654891
Train Epoch: 32 [10000/83967 (12%)]     Loss: 0.007926281541586
Train Epoch: 32 [15000/83967 (18%)]     Loss: 0.009008924476802
Train Epoch: 32 [20000/83967 (24%)]     Loss: 0.007618052419275
Train Epoch: 32 [25000/83967 (30%)]     Loss: 0.008020709268749
Train Epoch: 32 [30000/83967 (36%)]     Loss: 0.007400152739137
Train Epoch: 32 [35000/83967 (42%)]     Loss: 0.007994290441275
Train Epoch: 32 [40000/83967 (48%)]     Loss: 0.006565771531314
Train Epoch: 32 [45000/83967 (54%)]     Loss: 0.008539704605937
Train Epoch: 32 [50000/83967 (60%)]     Loss: 0.006846275646240
Train Epoch: 32 [55000/83967 (65%)]     Loss: 0.008406339213252
Train Epoch: 32 [60000/83967 (71%)]     Loss: 0.010190168395638
Train Epoch: 32 [65000/83967 (77%)]     Loss: 0.007146433927119
Train Epoch: 32 [70000/83967 (83%)]     Loss: 0.007746839895844
Train Epoch: 32 [75000/83967 (89%)]     Loss: 0.006816931534559
Train Epoch: 32 [80000/83967 (95%)]     Loss: 0.008216033689678
Train Epoch: 33 [0/83967 (0%)]  Loss: 0.008055783808231
Train Epoch: 33 [5000/83967 (6%)]       Loss: 0.007909867912531
Train Epoch: 33 [10000/83967 (12%)]     Loss: 0.008325629867613
Train Epoch: 33 [15000/83967 (18%)]     Loss: 0.007498339284211
Train Epoch: 33 [20000/83967 (24%)]     Loss: 0.007983624935150
Train Epoch: 33 [25000/83967 (30%)]     Loss: 0.007893763482571
Train Epoch: 33 [30000/83967 (36%)]     Loss: 0.007714009378105
Train Epoch: 33 [35000/83967 (42%)]     Loss: 0.009057288989425
Train Epoch: 33 [40000/83967 (48%)]     Loss: 0.008088257163763
Train Epoch: 33 [45000/83967 (54%)]     Loss: 0.006273066624999
Train Epoch: 33 [50000/83967 (60%)]     Loss: 0.007306249812245
Train Epoch: 33 [55000/83967 (65%)]     Loss: 0.008320277556777
Train Epoch: 33 [60000/83967 (71%)]     Loss: 0.008590949699283
Train Epoch: 33 [65000/83967 (77%)]     Loss: 0.008527750149369
Train Epoch: 33 [70000/83967 (83%)]     Loss: 0.008052540011704
Train Epoch: 33 [75000/83967 (89%)]     Loss: 0.009107760153711
Train Epoch: 33 [80000/83967 (95%)]     Loss: 0.008109236136079
Train Epoch: 34 [0/83967 (0%)]  Loss: 0.008351481519639
Train Epoch: 34 [5000/83967 (6%)]       Loss: 0.007205540314317
Train Epoch: 34 [10000/83967 (12%)]     Loss: 0.007549625821412
Train Epoch: 34 [15000/83967 (18%)]     Loss: 0.006951963994652
Train Epoch: 34 [20000/83967 (24%)]     Loss: 0.007065744604915
Train Epoch: 34 [25000/83967 (30%)]     Loss: 0.008730219677091
Train Epoch: 34 [30000/83967 (36%)]     Loss: 0.008801545947790
Train Epoch: 34 [35000/83967 (42%)]     Loss: 0.006530015729368
Train Epoch: 34 [40000/83967 (48%)]     Loss: 0.007126414682716
Train Epoch: 34 [45000/83967 (54%)]     Loss: 0.007248433306813
Train Epoch: 34 [50000/83967 (60%)]     Loss: 0.007976343855262
Train Epoch: 34 [55000/83967 (65%)]     Loss: 0.008437629789114
Train Epoch: 34 [60000/83967 (71%)]     Loss: 0.007921418175101
Train Epoch: 34 [65000/83967 (77%)]     Loss: 0.007676687091589
Train Epoch: 34 [70000/83967 (83%)]     Loss: 0.008065471425653
Train Epoch: 34 [75000/83967 (89%)]     Loss: 0.007648671045899
Train Epoch: 34 [80000/83967 (95%)]     Loss: 0.008886495605111
Train Epoch: 35 [0/83967 (0%)]  Loss: 0.008755401708186
Train Epoch: 35 [5000/83967 (6%)]       Loss: 0.008374528959394
Train Epoch: 35 [10000/83967 (12%)]     Loss: 0.008212740533054
Train Epoch: 35 [15000/83967 (18%)]     Loss: 0.008257118985057
Train Epoch: 35 [20000/83967 (24%)]     Loss: 0.008218404836953
Train Epoch: 35 [25000/83967 (30%)]     Loss: 0.007676023524255
Train Epoch: 35 [30000/83967 (36%)]     Loss: 0.008229334838688
Train Epoch: 35 [35000/83967 (42%)]     Loss: 0.006738672032952
Train Epoch: 35 [40000/83967 (48%)]     Loss: 0.007676779292524
Train Epoch: 35 [45000/83967 (54%)]     Loss: 0.006664827000350
Train Epoch: 35 [50000/83967 (60%)]     Loss: 0.008269915357232
Train Epoch: 35 [55000/83967 (65%)]     Loss: 0.008682920597494
Train Epoch: 35 [60000/83967 (71%)]     Loss: 0.006035097409040
Train Epoch: 35 [65000/83967 (77%)]     Loss: 0.008218485862017
Train Epoch: 35 [70000/83967 (83%)]     Loss: 0.007005578372627
Train Epoch: 35 [75000/83967 (89%)]     Loss: 0.008094707503915
Train Epoch: 35 [80000/83967 (95%)]     Loss: 0.007491922471672
Train Epoch: 36 [0/83967 (0%)]  Loss: 0.008602083660662
Train Epoch: 36 [5000/83967 (6%)]       Loss: 0.007771603297442
Train Epoch: 36 [10000/83967 (12%)]     Loss: 0.007690090220422
Train Epoch: 36 [15000/83967 (18%)]     Loss: 0.006864568218589
Train Epoch: 36 [20000/83967 (24%)]     Loss: 0.006894620135427
Train Epoch: 36 [25000/83967 (30%)]     Loss: 0.008521549403667
Train Epoch: 36 [30000/83967 (36%)]     Loss: 0.008314108476043
Train Epoch: 36 [35000/83967 (42%)]     Loss: 0.007698977831751
Train Epoch: 36 [40000/83967 (48%)]     Loss: 0.008546566590667
Train Epoch: 36 [45000/83967 (54%)]     Loss: 0.007816453464329
Train Epoch: 36 [50000/83967 (60%)]     Loss: 0.007886132225394
Train Epoch: 36 [55000/83967 (65%)]     Loss: 0.008116886019707
Train Epoch: 36 [60000/83967 (71%)]     Loss: 0.008171278983355
Train Epoch: 36 [65000/83967 (77%)]     Loss: 0.007400362752378
Train Epoch: 36 [70000/83967 (83%)]     Loss: 0.007127181626856
Train Epoch: 36 [75000/83967 (89%)]     Loss: 0.008936665952206
Train Epoch: 36 [80000/83967 (95%)]     Loss: 0.008290468715131
Train Epoch: 37 [0/83967 (0%)]  Loss: 0.008960997685790
Train Epoch: 37 [5000/83967 (6%)]       Loss: 0.007986353710294
Train Epoch: 37 [10000/83967 (12%)]     Loss: 0.006983500439674
Train Epoch: 37 [15000/83967 (18%)]     Loss: 0.007300880271941
Train Epoch: 37 [20000/83967 (24%)]     Loss: 0.009392002597451
Train Epoch: 37 [25000/83967 (30%)]     Loss: 0.008250800892711
Train Epoch: 37 [30000/83967 (36%)]     Loss: 0.008067345246673
Train Epoch: 37 [35000/83967 (42%)]     Loss: 0.006885370705277
Train Epoch: 37 [40000/83967 (48%)]     Loss: 0.007621895987540
Train Epoch: 37 [45000/83967 (54%)]     Loss: 0.007147150579840
Train Epoch: 37 [50000/83967 (60%)]     Loss: 0.008434873074293
Train Epoch: 37 [55000/83967 (65%)]     Loss: 0.007813477888703
Train Epoch: 37 [60000/83967 (71%)]     Loss: 0.006964591331780
Train Epoch: 37 [65000/83967 (77%)]     Loss: 0.007214581593871
Train Epoch: 37 [70000/83967 (83%)]     Loss: 0.006946885492653
Train Epoch: 37 [75000/83967 (89%)]     Loss: 0.008007752709091
Train Epoch: 37 [80000/83967 (95%)]     Loss: 0.006815237924457
Train Epoch: 38 [0/83967 (0%)]  Loss: 0.007430221885443
Train Epoch: 38 [5000/83967 (6%)]       Loss: 0.007172530982643
Train Epoch: 38 [10000/83967 (12%)]     Loss: 0.008176326751709
Train Epoch: 38 [15000/83967 (18%)]     Loss: 0.008225591853261
Train Epoch: 38 [20000/83967 (24%)]     Loss: 0.008694826625288
Train Epoch: 38 [25000/83967 (30%)]     Loss: 0.006993077695370
Train Epoch: 38 [30000/83967 (36%)]     Loss: 0.009247753769159
Train Epoch: 38 [35000/83967 (42%)]     Loss: 0.008445818908513
Train Epoch: 38 [40000/83967 (48%)]     Loss: 0.008483273908496
Train Epoch: 38 [45000/83967 (54%)]     Loss: 0.007261044345796
Train Epoch: 38 [50000/83967 (60%)]     Loss: 0.007344965357333
Train Epoch: 38 [55000/83967 (65%)]     Loss: 0.007595343515277
Train Epoch: 38 [60000/83967 (71%)]     Loss: 0.008028833195567
Train Epoch: 38 [65000/83967 (77%)]     Loss: 0.006719511933625
Train Epoch: 38 [70000/83967 (83%)]     Loss: 0.007486457936466
Train Epoch: 38 [75000/83967 (89%)]     Loss: 0.008177312090993
Train Epoch: 38 [80000/83967 (95%)]     Loss: 0.007230968214571
Train Epoch: 39 [0/83967 (0%)]  Loss: 0.007440533488989
Train Epoch: 39 [5000/83967 (6%)]       Loss: 0.007745092734694
Train Epoch: 39 [10000/83967 (12%)]     Loss: 0.007501808460802
Train Epoch: 39 [15000/83967 (18%)]     Loss: 0.007057574577630
Train Epoch: 39 [20000/83967 (24%)]     Loss: 0.009211050346494
Train Epoch: 39 [25000/83967 (30%)]     Loss: 0.008238133974373
Train Epoch: 39 [30000/83967 (36%)]     Loss: 0.008102120831609
Train Epoch: 39 [35000/83967 (42%)]     Loss: 0.008127982728183
Train Epoch: 39 [40000/83967 (48%)]     Loss: 0.007885225117207
Train Epoch: 39 [45000/83967 (54%)]     Loss: 0.007961295545101
Train Epoch: 39 [50000/83967 (60%)]     Loss: 0.008155015297234
Train Epoch: 39 [55000/83967 (65%)]     Loss: 0.008660721592605
Train Epoch: 39 [60000/83967 (71%)]     Loss: 0.008075484074652
Train Epoch: 39 [65000/83967 (77%)]     Loss: 0.007332836743444
Train Epoch: 39 [70000/83967 (83%)]     Loss: 0.007993679493666
Train Epoch: 39 [75000/83967 (89%)]     Loss: 0.007319594267756
Train Epoch: 39 [80000/83967 (95%)]     Loss: 0.007228549569845
Train Epoch: 40 [0/83967 (0%)]  Loss: 0.009136641398072
Train Epoch: 40 [5000/83967 (6%)]       Loss: 0.007116213906556
Train Epoch: 40 [10000/83967 (12%)]     Loss: 0.007109967991710
Train Epoch: 40 [15000/83967 (18%)]     Loss: 0.008268092758954
Train Epoch: 40 [20000/83967 (24%)]     Loss: 0.008682815358043
Train Epoch: 40 [25000/83967 (30%)]     Loss: 0.007920647971332
Train Epoch: 40 [30000/83967 (36%)]     Loss: 0.008873317390680
Train Epoch: 40 [35000/83967 (42%)]     Loss: 0.007967548444867
Train Epoch: 40 [40000/83967 (48%)]     Loss: 0.008068697527051
Train Epoch: 40 [45000/83967 (54%)]     Loss: 0.008022135123610
Train Epoch: 40 [50000/83967 (60%)]     Loss: 0.009238785132766
Train Epoch: 40 [55000/83967 (65%)]     Loss: 0.006878400687128
Train Epoch: 40 [60000/83967 (71%)]     Loss: 0.007119639776647
Train Epoch: 40 [65000/83967 (77%)]     Loss: 0.008778882212937
Train Epoch: 40 [70000/83967 (83%)]     Loss: 0.008126700296998
Train Epoch: 40 [75000/83967 (89%)]     Loss: 0.007398664951324
Train Epoch: 40 [80000/83967 (95%)]     Loss: 0.008521052077413
WARNING:tensorboardX.x2num:NaN or Inf found in input tensor.
Train Epoch: 41 [0/83967 (0%)]  Loss: 0.008504433557391
Train Epoch: 41 [5000/83967 (6%)]       Loss: 0.008398675359786
Train Epoch: 41 [10000/83967 (12%)]     Loss: 0.007112343795598
Train Epoch: 41 [15000/83967 (18%)]     Loss: 0.007514018099755
Train Epoch: 41 [20000/83967 (24%)]     Loss: 0.007460446096957
Train Epoch: 41 [25000/83967 (30%)]     Loss: 0.007767805363983
Train Epoch: 41 [30000/83967 (36%)]     Loss: 0.008018257096410
Train Epoch: 41 [35000/83967 (42%)]     Loss: 0.007808049209416
Train Epoch: 41 [40000/83967 (48%)]     Loss: 0.007764058187604
Train Epoch: 41 [45000/83967 (54%)]     Loss: 0.007250827737153
Train Epoch: 41 [50000/83967 (60%)]     Loss: 0.006824523210526
Train Epoch: 41 [55000/83967 (65%)]     Loss: 0.008180889301002
Train Epoch: 41 [60000/83967 (71%)]     Loss: 0.008692603558302
Train Epoch: 41 [65000/83967 (77%)]     Loss: 0.008384178392589
Train Epoch: 41 [70000/83967 (83%)]     Loss: 0.007710897363722
Train Epoch: 41 [75000/83967 (89%)]     Loss: 0.008317505009472
Train Epoch: 41 [80000/83967 (95%)]     Loss: 0.008577856235206
Train Epoch: 42 [0/83967 (0%)]  Loss: 0.008069841191173
Train Epoch: 42 [5000/83967 (6%)]       Loss: 0.009198969230056
Train Epoch: 42 [10000/83967 (12%)]     Loss: 0.008518859744072
Train Epoch: 42 [15000/83967 (18%)]     Loss: 0.008223468437791
Train Epoch: 42 [20000/83967 (24%)]     Loss: 0.007941510528326
Train Epoch: 42 [25000/83967 (30%)]     Loss: 0.007215227000415
Train Epoch: 42 [30000/83967 (36%)]     Loss: 0.007516656070948
Train Epoch: 42 [35000/83967 (42%)]     Loss: 0.007512847427279
Train Epoch: 42 [40000/83967 (48%)]     Loss: 0.008450858294964
Train Epoch: 42 [45000/83967 (54%)]     Loss: 0.007421492598951
Train Epoch: 42 [50000/83967 (60%)]     Loss: 0.007672009523958
Train Epoch: 42 [55000/83967 (65%)]     Loss: 0.007756234146655
Train Epoch: 42 [60000/83967 (71%)]     Loss: 0.008444690145552
Train Epoch: 42 [65000/83967 (77%)]     Loss: 0.008184837177396
Train Epoch: 42 [70000/83967 (83%)]     Loss: 0.008396090939641
Train Epoch: 42 [75000/83967 (89%)]     Loss: 0.007606449536979
Train Epoch: 42 [80000/83967 (95%)]     Loss: 0.008699564263225
Train Epoch: 43 [0/83967 (0%)]  Loss: 0.007877858355641
Train Epoch: 43 [5000/83967 (6%)]       Loss: 0.008423347026110
Train Epoch: 43 [10000/83967 (12%)]     Loss: 0.008268596604466
Train Epoch: 43 [15000/83967 (18%)]     Loss: 0.007354878354818
Train Epoch: 43 [20000/83967 (24%)]     Loss: 0.008087900467217
Train Epoch: 43 [25000/83967 (30%)]     Loss: 0.007020197343081
Train Epoch: 43 [30000/83967 (36%)]     Loss: 0.007742781192064
Train Epoch: 43 [35000/83967 (42%)]     Loss: 0.007322475779802
Train Epoch: 43 [40000/83967 (48%)]     Loss: 0.007943538017571
Train Epoch: 43 [45000/83967 (54%)]     Loss: 0.007960362359881
Train Epoch: 43 [50000/83967 (60%)]     Loss: 0.007124126888812
Train Epoch: 43 [55000/83967 (65%)]     Loss: 0.007102666422725
Train Epoch: 43 [60000/83967 (71%)]     Loss: 0.008510448969901
Train Epoch: 43 [65000/83967 (77%)]     Loss: 0.008048306219280
Train Epoch: 43 [70000/83967 (83%)]     Loss: 0.008362806402147
Train Epoch: 43 [75000/83967 (89%)]     Loss: 0.007307421416044
Train Epoch: 43 [80000/83967 (95%)]     Loss: 0.006979434750974
Train Epoch: 44 [0/83967 (0%)]  Loss: 0.008844682015479
Train Epoch: 44 [5000/83967 (6%)]       Loss: 0.008839627727866
Train Epoch: 44 [10000/83967 (12%)]     Loss: 0.006895841099322
Train Epoch: 44 [15000/83967 (18%)]     Loss: 0.008576510474086
Train Epoch: 44 [20000/83967 (24%)]     Loss: 0.007859090343118
Train Epoch: 44 [25000/83967 (30%)]     Loss: 0.007187754847109
Train Epoch: 44 [30000/83967 (36%)]     Loss: 0.007831801660359
Train Epoch: 44 [35000/83967 (42%)]     Loss: 0.007462730631232
Train Epoch: 44 [40000/83967 (48%)]     Loss: 0.008305946364999
Train Epoch: 44 [45000/83967 (54%)]     Loss: 0.007236381061375
Train Epoch: 44 [50000/83967 (60%)]     Loss: 0.008185322396457
Train Epoch: 44 [55000/83967 (65%)]     Loss: 0.007354121655226
Train Epoch: 44 [60000/83967 (71%)]     Loss: 0.007897214964032
Train Epoch: 44 [65000/83967 (77%)]     Loss: 0.007334113586694
Train Epoch: 44 [70000/83967 (83%)]     Loss: 0.007960450835526
Train Epoch: 44 [75000/83967 (89%)]     Loss: 0.007501489482820
Train Epoch: 44 [80000/83967 (95%)]     Loss: 0.007975297048688
WARNING:tensorboardX.x2num:NaN or Inf found in input tensor.
Train Epoch: 45 [0/83967 (0%)]  Loss: 0.006597014609724
Train Epoch: 45 [5000/83967 (6%)]       Loss: 0.008514812216163
Train Epoch: 45 [10000/83967 (12%)]     Loss: 0.008643716573715
Train Epoch: 45 [15000/83967 (18%)]     Loss: 0.007323834113777
Train Epoch: 45 [20000/83967 (24%)]     Loss: 0.008476670831442
Train Epoch: 45 [25000/83967 (30%)]     Loss: 0.008805537596345
Train Epoch: 45 [30000/83967 (36%)]     Loss: 0.008670106530190
Train Epoch: 45 [35000/83967 (42%)]     Loss: 0.007530106697232
Train Epoch: 45 [40000/83967 (48%)]     Loss: 0.007796956226230
Train Epoch: 45 [45000/83967 (54%)]     Loss: 0.008021553978324
Train Epoch: 45 [50000/83967 (60%)]     Loss: 0.008892844431102
Train Epoch: 45 [55000/83967 (65%)]     Loss: 0.007039000280201
Train Epoch: 45 [60000/83967 (71%)]     Loss: 0.008235352113843
Train Epoch: 45 [65000/83967 (77%)]     Loss: 0.008772210218012
Train Epoch: 45 [70000/83967 (83%)]     Loss: 0.006572952028364
Train Epoch: 45 [75000/83967 (89%)]     Loss: 0.007826907560229
Train Epoch: 45 [80000/83967 (95%)]     Loss: 0.008099294267595
Train Epoch: 46 [0/83967 (0%)]  Loss: 0.006320067215711
Train Epoch: 46 [5000/83967 (6%)]       Loss: 0.007969344966114
Train Epoch: 46 [10000/83967 (12%)]     Loss: 0.008713770657778
Train Epoch: 46 [15000/83967 (18%)]     Loss: 0.006312246434391
Train Epoch: 46 [20000/83967 (24%)]     Loss: 0.007987612858415
Train Epoch: 46 [25000/83967 (30%)]     Loss: 0.007719465065747
Train Epoch: 46 [30000/83967 (36%)]     Loss: 0.006765542551875
Train Epoch: 46 [35000/83967 (42%)]     Loss: 0.008555579930544
Train Epoch: 46 [40000/83967 (48%)]     Loss: 0.007114692591131
Train Epoch: 46 [45000/83967 (54%)]     Loss: 0.006716502830386
Train Epoch: 46 [50000/83967 (60%)]     Loss: 0.006933128461242
Train Epoch: 46 [55000/83967 (65%)]     Loss: 0.007749342359602
Train Epoch: 46 [60000/83967 (71%)]     Loss: 0.007789881899953
Train Epoch: 46 [65000/83967 (77%)]     Loss: 0.007278956472874
Train Epoch: 46 [70000/83967 (83%)]     Loss: 0.008653938770294
Train Epoch: 46 [75000/83967 (89%)]     Loss: 0.008633309975266
Train Epoch: 46 [80000/83967 (95%)]     Loss: 0.006866258569062
Train Epoch: 47 [0/83967 (0%)]  Loss: 0.007942215539515
Train Epoch: 47 [5000/83967 (6%)]       Loss: 0.007108743302524
Train Epoch: 47 [10000/83967 (12%)]     Loss: 0.007600560784340
Train Epoch: 47 [15000/83967 (18%)]     Loss: 0.006408078130335
Train Epoch: 47 [20000/83967 (24%)]     Loss: 0.007800946943462
Train Epoch: 47 [25000/83967 (30%)]     Loss: 0.007915858179331
Train Epoch: 47 [30000/83967 (36%)]     Loss: 0.007665359415114
Train Epoch: 47 [35000/83967 (42%)]     Loss: 0.008178097195923
Train Epoch: 47 [40000/83967 (48%)]     Loss: 0.006884955335408
Train Epoch: 47 [45000/83967 (54%)]     Loss: 0.008588831871748
Train Epoch: 47 [50000/83967 (60%)]     Loss: 0.007919544354081
Train Epoch: 47 [55000/83967 (65%)]     Loss: 0.008772675879300
Train Epoch: 47 [60000/83967 (71%)]     Loss: 0.008952319622040
Train Epoch: 47 [65000/83967 (77%)]     Loss: 0.007338272873312
Train Epoch: 47 [70000/83967 (83%)]     Loss: 0.008167427964509
Train Epoch: 47 [75000/83967 (89%)]     Loss: 0.008512485772371
Train Epoch: 47 [80000/83967 (95%)]     Loss: 0.008105947636068
Train Epoch: 48 [0/83967 (0%)]  Loss: 0.007074762601405
Train Epoch: 48 [5000/83967 (6%)]       Loss: 0.008063349872828
Train Epoch: 48 [10000/83967 (12%)]     Loss: 0.007921323180199
Train Epoch: 48 [15000/83967 (18%)]     Loss: 0.008535645902157
Train Epoch: 48 [20000/83967 (24%)]     Loss: 0.008277444168925
Train Epoch: 48 [25000/83967 (30%)]     Loss: 0.007198810111731
Train Epoch: 48 [30000/83967 (36%)]     Loss: 0.008810560218990
Train Epoch: 48 [35000/83967 (42%)]     Loss: 0.007708211895078
Train Epoch: 48 [40000/83967 (48%)]     Loss: 0.007312390021980
Train Epoch: 48 [45000/83967 (54%)]     Loss: 0.008345231413841
Train Epoch: 48 [50000/83967 (60%)]     Loss: 0.007094104308635
Train Epoch: 48 [55000/83967 (65%)]     Loss: 0.007758657447994
Train Epoch: 48 [60000/83967 (71%)]     Loss: 0.009587205946445
Train Epoch: 48 [65000/83967 (77%)]     Loss: 0.007578144315630
Train Epoch: 48 [70000/83967 (83%)]     Loss: 0.007509815040976
Train Epoch: 48 [75000/83967 (89%)]     Loss: 0.006672854535282
Train Epoch: 48 [80000/83967 (95%)]     Loss: 0.007373285014182
WARNING:tensorboardX.x2num:NaN or Inf found in input tensor.
Train Epoch: 49 [0/83967 (0%)]  Loss: 0.007179862819612
Train Epoch: 49 [5000/83967 (6%)]       Loss: 0.007845148444176
Train Epoch: 49 [10000/83967 (12%)]     Loss: 0.006874734535813
Train Epoch: 49 [15000/83967 (18%)]     Loss: 0.008650593459606
Train Epoch: 49 [20000/83967 (24%)]     Loss: 0.008413083851337
Train Epoch: 49 [25000/83967 (30%)]     Loss: 0.007524684537202
Train Epoch: 49 [30000/83967 (36%)]     Loss: 0.008314343169332
Train Epoch: 49 [35000/83967 (42%)]     Loss: 0.008105855435133
Train Epoch: 49 [40000/83967 (48%)]     Loss: 0.007223600987345
Train Epoch: 49 [45000/83967 (54%)]     Loss: 0.007806864567101
Train Epoch: 49 [50000/83967 (60%)]     Loss: 0.007752478122711
Train Epoch: 49 [55000/83967 (65%)]     Loss: 0.007924779318273
Train Epoch: 49 [60000/83967 (71%)]     Loss: 0.006638427730650
Train Epoch: 49 [65000/83967 (77%)]     Loss: 0.006881626788527
Train Epoch: 49 [70000/83967 (83%)]     Loss: 0.007852783426642
Train Epoch: 49 [75000/83967 (89%)]     Loss: 0.006787416059524
Train Epoch: 49 [80000/83967 (95%)]     Loss: 0.008513203822076
Train Epoch: 50 [0/83967 (0%)]  Loss: 0.007724591530859
Train Epoch: 50 [5000/83967 (6%)]       Loss: 0.007517844438553
Train Epoch: 50 [10000/83967 (12%)]     Loss: 0.007804019376636
Train Epoch: 50 [15000/83967 (18%)]     Loss: 0.007895046845078
Train Epoch: 50 [20000/83967 (24%)]     Loss: 0.008256703615189
Train Epoch: 50 [25000/83967 (30%)]     Loss: 0.007805006578565
Train Epoch: 50 [30000/83967 (36%)]     Loss: 0.006777092348784
Train Epoch: 50 [35000/83967 (42%)]     Loss: 0.007124391850084
Train Epoch: 50 [40000/83967 (48%)]     Loss: 0.007776909042150
Train Epoch: 50 [45000/83967 (54%)]     Loss: 0.007091295439750
Train Epoch: 50 [50000/83967 (60%)]     Loss: 0.007789452560246
Train Epoch: 50 [55000/83967 (65%)]     Loss: 0.007591606117785
Train Epoch: 50 [60000/83967 (71%)]     Loss: 0.007972656749189
Train Epoch: 50 [65000/83967 (77%)]     Loss: 0.007456125225872
Train Epoch: 50 [70000/83967 (83%)]     Loss: 0.008114636875689
Train Epoch: 50 [75000/83967 (89%)]     Loss: 0.007347075268626
Train Epoch: 50 [80000/83967 (95%)]     Loss: 0.008169556036592
Train Epoch: 51 [0/83967 (0%)]  Loss: 0.007260236889124
Train Epoch: 51 [5000/83967 (6%)]       Loss: 0.006719865836203
Train Epoch: 51 [10000/83967 (12%)]     Loss: 0.007584657520056
Train Epoch: 51 [15000/83967 (18%)]     Loss: 0.007841875776649
Train Epoch: 51 [20000/83967 (24%)]     Loss: 0.007498370017856
Train Epoch: 51 [25000/83967 (30%)]     Loss: 0.006762784440070
Train Epoch: 51 [30000/83967 (36%)]     Loss: 0.006729453802109
Train Epoch: 51 [35000/83967 (42%)]     Loss: 0.008209124207497
Train Epoch: 51 [40000/83967 (48%)]     Loss: 0.006846604403108
Train Epoch: 51 [45000/83967 (54%)]     Loss: 0.008057623170316
Train Epoch: 51 [50000/83967 (60%)]     Loss: 0.008314017206430
Train Epoch: 51 [55000/83967 (65%)]     Loss: 0.006741587072611
Train Epoch: 51 [60000/83967 (71%)]     Loss: 0.007170671597123
Train Epoch: 51 [65000/83967 (77%)]     Loss: 0.007242738269269
Train Epoch: 51 [70000/83967 (83%)]     Loss: 0.008071567863226
Train Epoch: 51 [75000/83967 (89%)]     Loss: 0.006890697870404
Train Epoch: 51 [80000/83967 (95%)]     Loss: 0.008824064396322
WARNING:tensorboardX.x2num:NaN or Inf found in input tensor.
Train Epoch: 52 [0/83967 (0%)]  Loss: 0.008311345241964
Train Epoch: 52 [5000/83967 (6%)]       Loss: 0.008549867197871
Train Epoch: 52 [10000/83967 (12%)]     Loss: 0.007816085591912
Train Epoch: 52 [15000/83967 (18%)]     Loss: 0.006824246607721
Train Epoch: 52 [20000/83967 (24%)]     Loss: 0.009621223434806
Train Epoch: 52 [25000/83967 (30%)]     Loss: 0.007275560405105
Train Epoch: 52 [30000/83967 (36%)]     Loss: 0.007524263579398
Train Epoch: 52 [35000/83967 (42%)]     Loss: 0.006348282098770
Train Epoch: 52 [40000/83967 (48%)]     Loss: 0.008616333827376
Train Epoch: 52 [45000/83967 (54%)]     Loss: 0.007080397568643
Train Epoch: 52 [50000/83967 (60%)]     Loss: 0.007604978512973
Train Epoch: 52 [55000/83967 (65%)]     Loss: 0.007846665568650
Train Epoch: 52 [60000/83967 (71%)]     Loss: 0.007622580043972
Train Epoch: 52 [65000/83967 (77%)]     Loss: 0.007207492366433
Train Epoch: 52 [70000/83967 (83%)]     Loss: 0.006648539099842
Train Epoch: 52 [75000/83967 (89%)]     Loss: 0.007425549440086
Train Epoch: 52 [80000/83967 (95%)]     Loss: 0.009073251858354
Train Epoch: 53 [0/83967 (0%)]  Loss: 0.007903904654086
Train Epoch: 53 [5000/83967 (6%)]       Loss: 0.005324266850948
Train Epoch: 53 [10000/83967 (12%)]     Loss: 0.008276259526610
Train Epoch: 53 [15000/83967 (18%)]     Loss: 0.006846119649708
Train Epoch: 53 [20000/83967 (24%)]     Loss: 0.008732717484236
Train Epoch: 53 [25000/83967 (30%)]     Loss: 0.007840544916689
Train Epoch: 53 [30000/83967 (36%)]     Loss: 0.007858541794121
Train Epoch: 53 [35000/83967 (42%)]     Loss: 0.006260298192501
Train Epoch: 53 [40000/83967 (48%)]     Loss: 0.007776725105941
Train Epoch: 53 [45000/83967 (54%)]     Loss: 0.006402391009033
Train Epoch: 53 [50000/83967 (60%)]     Loss: 0.007720431778580
Train Epoch: 53 [55000/83967 (65%)]     Loss: 0.008537851274014
Train Epoch: 53 [60000/83967 (71%)]     Loss: 0.007158691529185
Train Epoch: 53 [65000/83967 (77%)]     Loss: 0.008068979717791
Train Epoch: 53 [70000/83967 (83%)]     Loss: 0.007572723552585
Train Epoch: 53 [75000/83967 (89%)]     Loss: 0.007911728695035
Train Epoch: 53 [80000/83967 (95%)]     Loss: 0.007865991443396
Train Epoch: 54 [0/83967 (0%)]  Loss: 0.006774119101465
Train Epoch: 54 [5000/83967 (6%)]       Loss: 0.008742005564272
Train Epoch: 54 [10000/83967 (12%)]     Loss: 0.008220917545259
Train Epoch: 54 [15000/83967 (18%)]     Loss: 0.007232770323753
Train Epoch: 54 [20000/83967 (24%)]     Loss: 0.008150992915034
Train Epoch: 54 [25000/83967 (30%)]     Loss: 0.006033128593117
Train Epoch: 54 [30000/83967 (36%)]     Loss: 0.009333049878478
Train Epoch: 54 [35000/83967 (42%)]     Loss: 0.008214244619012
Train Epoch: 54 [40000/83967 (48%)]     Loss: 0.007052103988826
Train Epoch: 54 [45000/83967 (54%)]     Loss: 0.007167027331889
Train Epoch: 54 [50000/83967 (60%)]     Loss: 0.007606651168317
Train Epoch: 54 [55000/83967 (65%)]     Loss: 0.007347718346864
Train Epoch: 54 [60000/83967 (71%)]     Loss: 0.006195428781211
Train Epoch: 54 [65000/83967 (77%)]     Loss: 0.007987290620804
Train Epoch: 54 [70000/83967 (83%)]     Loss: 0.008670862764120
Train Epoch: 54 [75000/83967 (89%)]     Loss: 0.008292373269796
Train Epoch: 54 [80000/83967 (95%)]     Loss: 0.008177733048797
Train Epoch: 55 [0/83967 (0%)]  Loss: 0.008519346825778
Train Epoch: 55 [5000/83967 (6%)]       Loss: 0.007148799952120
Train Epoch: 55 [10000/83967 (12%)]     Loss: 0.006703308783472
Train Epoch: 55 [15000/83967 (18%)]     Loss: 0.007808345835656
Train Epoch: 55 [20000/83967 (24%)]     Loss: 0.008132684044540
Train Epoch: 55 [25000/83967 (30%)]     Loss: 0.007915904745460
Train Epoch: 55 [30000/83967 (36%)]     Loss: 0.007979871705174
Train Epoch: 55 [35000/83967 (42%)]     Loss: 0.006707396358252
Train Epoch: 55 [40000/83967 (48%)]     Loss: 0.007599613629282
Train Epoch: 55 [45000/83967 (54%)]     Loss: 0.007455146405846
Train Epoch: 55 [50000/83967 (60%)]     Loss: 0.007297195959836
Train Epoch: 55 [55000/83967 (65%)]     Loss: 0.008500430732965
Train Epoch: 55 [60000/83967 (71%)]     Loss: 0.008422290906310
Train Epoch: 55 [65000/83967 (77%)]     Loss: 0.007714812178165
Train Epoch: 55 [70000/83967 (83%)]     Loss: 0.008190519176424
Train Epoch: 55 [75000/83967 (89%)]     Loss: 0.007637646980584
Train Epoch: 55 [80000/83967 (95%)]     Loss: 0.007670907303691
Train Epoch: 56 [0/83967 (0%)]  Loss: 0.007944998331368
Train Epoch: 56 [5000/83967 (6%)]       Loss: 0.006312752142549
Train Epoch: 56 [10000/83967 (12%)]     Loss: 0.009394774213433
Train Epoch: 56 [15000/83967 (18%)]     Loss: 0.006899575237185
Train Epoch: 56 [20000/83967 (24%)]     Loss: 0.007671835832298
Train Epoch: 56 [25000/83967 (30%)]     Loss: 0.007441601715982
Train Epoch: 56 [30000/83967 (36%)]     Loss: 0.007857450284064
Train Epoch: 56 [35000/83967 (42%)]     Loss: 0.007138181943446
Train Epoch: 56 [40000/83967 (48%)]     Loss: 0.007665821816772
Train Epoch: 56 [45000/83967 (54%)]     Loss: 0.007829881273210
Train Epoch: 56 [50000/83967 (60%)]     Loss: 0.007453260477632
Train Epoch: 56 [55000/83967 (65%)]     Loss: 0.006847810000181
Train Epoch: 56 [60000/83967 (71%)]     Loss: 0.007576397620142
Train Epoch: 56 [65000/83967 (77%)]     Loss: 0.008244036696851
Train Epoch: 56 [70000/83967 (83%)]     Loss: 0.007557387929410
Train Epoch: 56 [75000/83967 (89%)]     Loss: 0.007143073715270
Train Epoch: 56 [80000/83967 (95%)]     Loss: 0.008733524940908
Train Epoch: 57 [0/83967 (0%)]  Loss: 0.007698277477175
Train Epoch: 57 [5000/83967 (6%)]       Loss: 0.006592394784093
Train Epoch: 57 [10000/83967 (12%)]     Loss: 0.006918169558048
Train Epoch: 57 [15000/83967 (18%)]     Loss: 0.006191740278155
Train Epoch: 57 [20000/83967 (24%)]     Loss: 0.007669048383832
Train Epoch: 57 [25000/83967 (30%)]     Loss: 0.008448703214526
Train Epoch: 57 [30000/83967 (36%)]     Loss: 0.007142359390855
Train Epoch: 57 [35000/83967 (42%)]     Loss: 0.006632107309997
Train Epoch: 57 [40000/83967 (48%)]     Loss: 0.007588184438646
Train Epoch: 57 [45000/83967 (54%)]     Loss: 0.007084958255291
Train Epoch: 57 [50000/83967 (60%)]     Loss: 0.006471187807620
Train Epoch: 57 [55000/83967 (65%)]     Loss: 0.007948311045766
Train Epoch: 57 [60000/83967 (71%)]     Loss: 0.007031950633973
Train Epoch: 57 [65000/83967 (77%)]     Loss: 0.007648051250726
Train Epoch: 57 [70000/83967 (83%)]     Loss: 0.006747687701136
Train Epoch: 57 [75000/83967 (89%)]     Loss: 0.007697614841163
Train Epoch: 57 [80000/83967 (95%)]     Loss: 0.007416564039886
Train Epoch: 58 [0/83967 (0%)]  Loss: 0.006534739863127
Train Epoch: 58 [5000/83967 (6%)]       Loss: 0.007225561887026
Train Epoch: 58 [10000/83967 (12%)]     Loss: 0.007789352908731
Train Epoch: 58 [15000/83967 (18%)]     Loss: 0.007002170663327
Train Epoch: 58 [20000/83967 (24%)]     Loss: 0.007590747904032
Train Epoch: 58 [25000/83967 (30%)]     Loss: 0.007458158768713
Train Epoch: 58 [30000/83967 (36%)]     Loss: 0.006927974522114
Train Epoch: 58 [35000/83967 (42%)]     Loss: 0.007441584952176
Train Epoch: 58 [40000/83967 (48%)]     Loss: 0.007408523466438
Train Epoch: 58 [45000/83967 (54%)]     Loss: 0.008310119621456
Train Epoch: 58 [50000/83967 (60%)]     Loss: 0.006564179435372
Train Epoch: 58 [55000/83967 (65%)]     Loss: 0.008434318006039
Train Epoch: 58 [60000/83967 (71%)]     Loss: 0.007497083861381
Train Epoch: 58 [65000/83967 (77%)]     Loss: 0.008496197871864
Train Epoch: 58 [70000/83967 (83%)]     Loss: 0.007971701212227
Train Epoch: 58 [75000/83967 (89%)]     Loss: 0.008507867343724
Train Epoch: 58 [80000/83967 (95%)]     Loss: 0.007892999798059
Train Epoch: 59 [0/83967 (0%)]  Loss: 0.008323853835464
Train Epoch: 59 [5000/83967 (6%)]       Loss: 0.007160411681980
Train Epoch: 59 [10000/83967 (12%)]     Loss: 0.008766790851951
Train Epoch: 59 [15000/83967 (18%)]     Loss: 0.006764300633222
Train Epoch: 59 [20000/83967 (24%)]     Loss: 0.006640155334026
Train Epoch: 59 [25000/83967 (30%)]     Loss: 0.009061588905752
Train Epoch: 59 [30000/83967 (36%)]     Loss: 0.006375921424478
Train Epoch: 59 [35000/83967 (42%)]     Loss: 0.009253188967705
Train Epoch: 59 [40000/83967 (48%)]     Loss: 0.007267523091286
Train Epoch: 59 [45000/83967 (54%)]     Loss: 0.007039242889732
Train Epoch: 59 [50000/83967 (60%)]     Loss: 0.007664142176509
Train Epoch: 59 [55000/83967 (65%)]     Loss: 0.008351640775800
Train Epoch: 59 [60000/83967 (71%)]     Loss: 0.008490247651935
Train Epoch: 59 [65000/83967 (77%)]     Loss: 0.007769823074341
Train Epoch: 59 [70000/83967 (83%)]     Loss: 0.006661415100098
Train Epoch: 59 [75000/83967 (89%)]     Loss: 0.008668798953295
Train Epoch: 59 [80000/83967 (95%)]     Loss: 0.007609450258315
Train Epoch: 60 [0/83967 (0%)]  Loss: 0.008784496225417
Train Epoch: 60 [5000/83967 (6%)]       Loss: 0.005883138626814
Train Epoch: 60 [10000/83967 (12%)]     Loss: 0.007563621737063
Train Epoch: 60 [15000/83967 (18%)]     Loss: 0.007724861614406
Train Epoch: 60 [20000/83967 (24%)]     Loss: 0.007455365732312
Train Epoch: 60 [25000/83967 (30%)]     Loss: 0.007666321005672
Train Epoch: 60 [30000/83967 (36%)]     Loss: 0.006639453582466
Train Epoch: 60 [35000/83967 (42%)]     Loss: 0.007841554470360
Train Epoch: 60 [40000/83967 (48%)]     Loss: 0.008183242753148
Train Epoch: 60 [45000/83967 (54%)]     Loss: 0.008338802494109
Train Epoch: 60 [50000/83967 (60%)]     Loss: 0.007329039741307
Train Epoch: 60 [55000/83967 (65%)]     Loss: 0.007040006108582
Train Epoch: 60 [60000/83967 (71%)]     Loss: 0.008038095198572
Train Epoch: 60 [65000/83967 (77%)]     Loss: 0.007939062081277
Train Epoch: 60 [70000/83967 (83%)]     Loss: 0.007446637377143
Train Epoch: 60 [75000/83967 (89%)]     Loss: 0.008062224835157
Train Epoch: 60 [80000/83967 (95%)]     Loss: 0.006446671672165
Train Epoch: 61 [0/83967 (0%)]  Loss: 0.008594547398388
Train Epoch: 61 [5000/83967 (6%)]       Loss: 0.007110401056707
Train Epoch: 61 [10000/83967 (12%)]     Loss: 0.008480501361191
Train Epoch: 61 [15000/83967 (18%)]     Loss: 0.007953828200698
Train Epoch: 61 [20000/83967 (24%)]     Loss: 0.006848017219454
Train Epoch: 61 [25000/83967 (30%)]     Loss: 0.007408134173602
Train Epoch: 61 [30000/83967 (36%)]     Loss: 0.006985726300627
Train Epoch: 61 [35000/83967 (42%)]     Loss: 0.008514017798007
Train Epoch: 61 [40000/83967 (48%)]     Loss: 0.008139943704009
Train Epoch: 61 [45000/83967 (54%)]     Loss: 0.007210740819573
Train Epoch: 61 [50000/83967 (60%)]     Loss: 0.007336139678955
Train Epoch: 61 [55000/83967 (65%)]     Loss: 0.007093667984009
Train Epoch: 61 [60000/83967 (71%)]     Loss: 0.008129901252687
Train Epoch: 61 [65000/83967 (77%)]     Loss: 0.006720406934619
Train Epoch: 61 [70000/83967 (83%)]     Loss: 0.008206657133996
Train Epoch: 61 [75000/83967 (89%)]     Loss: 0.008078450337052
Train Epoch: 61 [80000/83967 (95%)]     Loss: 0.008853769861162
Train Epoch: 62 [0/83967 (0%)]  Loss: 0.008060703985393
Train Epoch: 62 [5000/83967 (6%)]       Loss: 0.008031030185521
Train Epoch: 62 [10000/83967 (12%)]     Loss: 0.008616262115538
Train Epoch: 62 [15000/83967 (18%)]     Loss: 0.008242666721344
Train Epoch: 62 [20000/83967 (24%)]     Loss: 0.008281107991934
Train Epoch: 62 [25000/83967 (30%)]     Loss: 0.008311564102769
Train Epoch: 62 [30000/83967 (36%)]     Loss: 0.007719152141362
Train Epoch: 62 [35000/83967 (42%)]     Loss: 0.007704537361860
Train Epoch: 62 [40000/83967 (48%)]     Loss: 0.006829875521362
Train Epoch: 62 [45000/83967 (54%)]     Loss: 0.008399500511587
Train Epoch: 62 [50000/83967 (60%)]     Loss: 0.006865217350423
Train Epoch: 62 [55000/83967 (65%)]     Loss: 0.007359996903688
Train Epoch: 62 [60000/83967 (71%)]     Loss: 0.007709172554314
Train Epoch: 62 [65000/83967 (77%)]     Loss: 0.007856138050556
Train Epoch: 62 [70000/83967 (83%)]     Loss: 0.007323822937906
Train Epoch: 62 [75000/83967 (89%)]     Loss: 0.006491491571069
Train Epoch: 62 [80000/83967 (95%)]     Loss: 0.006222785450518
Train Epoch: 63 [0/83967 (0%)]  Loss: 0.008494467474520
Train Epoch: 63 [5000/83967 (6%)]       Loss: 0.006911414209753
Train Epoch: 63 [10000/83967 (12%)]     Loss: 0.008777436800301
Train Epoch: 63 [15000/83967 (18%)]     Loss: 0.009040918201208
Train Epoch: 63 [20000/83967 (24%)]     Loss: 0.008144043385983
Train Epoch: 63 [25000/83967 (30%)]     Loss: 0.007274317555130
Train Epoch: 63 [30000/83967 (36%)]     Loss: 0.006907766684890
Train Epoch: 63 [35000/83967 (42%)]     Loss: 0.008072795346379
Train Epoch: 63 [40000/83967 (48%)]     Loss: 0.006558829918504
Train Epoch: 63 [45000/83967 (54%)]     Loss: 0.007250507362187
Train Epoch: 63 [50000/83967 (60%)]     Loss: 0.008023292757571
Train Epoch: 63 [55000/83967 (65%)]     Loss: 0.006939882412553
Train Epoch: 63 [60000/83967 (71%)]     Loss: 0.007508433423936
Train Epoch: 63 [65000/83967 (77%)]     Loss: 0.006748311221600
Train Epoch: 63 [70000/83967 (83%)]     Loss: 0.007241011597216
Train Epoch: 63 [75000/83967 (89%)]     Loss: 0.006376920267940
Train Epoch: 63 [80000/83967 (95%)]     Loss: 0.007447130046785
Train Epoch: 64 [0/83967 (0%)]  Loss: 0.006825332995504
Train Epoch: 64 [5000/83967 (6%)]       Loss: 0.007498257793486
Train Epoch: 64 [10000/83967 (12%)]     Loss: 0.007087165489793
Train Epoch: 64 [15000/83967 (18%)]     Loss: 0.008372644893825
Train Epoch: 64 [20000/83967 (24%)]     Loss: 0.007316475734115
Train Epoch: 64 [25000/83967 (30%)]     Loss: 0.008063877932727
Train Epoch: 64 [30000/83967 (36%)]     Loss: 0.005646435543895
Train Epoch: 64 [35000/83967 (42%)]     Loss: 0.007370310369879
Train Epoch: 64 [40000/83967 (48%)]     Loss: 0.008718369528651
Train Epoch: 64 [45000/83967 (54%)]     Loss: 0.008565752767026
Train Epoch: 64 [50000/83967 (60%)]     Loss: 0.008101568557322
Train Epoch: 64 [55000/83967 (65%)]     Loss: 0.007600054144859
Train Epoch: 64 [60000/83967 (71%)]     Loss: 0.008445779792964
Train Epoch: 64 [65000/83967 (77%)]     Loss: 0.007697256747633
Train Epoch: 64 [70000/83967 (83%)]     Loss: 0.007686397992074
Train Epoch: 64 [75000/83967 (89%)]     Loss: 0.007055129390210
Train Epoch: 64 [80000/83967 (95%)]     Loss: 0.007112737279385
Train Epoch: 65 [0/83967 (0%)]  Loss: 0.007651983760297
Train Epoch: 65 [5000/83967 (6%)]       Loss: 0.007277786731720
Train Epoch: 65 [10000/83967 (12%)]     Loss: 0.008057784289122
Train Epoch: 65 [15000/83967 (18%)]     Loss: 0.007852782495320
Train Epoch: 65 [20000/83967 (24%)]     Loss: 0.007591740693897
Train Epoch: 65 [25000/83967 (30%)]     Loss: 0.006606060080230
Train Epoch: 65 [30000/83967 (36%)]     Loss: 0.007268037647009
Train Epoch: 65 [35000/83967 (42%)]     Loss: 0.007364076562226
Train Epoch: 65 [40000/83967 (48%)]     Loss: 0.008785249665380
Train Epoch: 65 [45000/83967 (54%)]     Loss: 0.007727322634310
Train Epoch: 65 [50000/83967 (60%)]     Loss: 0.007666467688978
Train Epoch: 65 [55000/83967 (65%)]     Loss: 0.007856193929911
Train Epoch: 65 [60000/83967 (71%)]     Loss: 0.007984674535692
Train Epoch: 65 [65000/83967 (77%)]     Loss: 0.007068237289786
Train Epoch: 65 [70000/83967 (83%)]     Loss: 0.006213417742401
Train Epoch: 65 [75000/83967 (89%)]     Loss: 0.007138627581298

The loss seems to be stuck around the 0.007 value after approximately 200 epochs. Is it still decreasing?
I am concerned because the results are far from the obtained in the paper.

Some inferences with my training:

Some beliefs with my training:
00000_belief:

00100_belief:

00200_belief:

00300_belief:

What can I do to improve this result, any suggestion?
Thanks in advance 💪

[TontonTremblay]

I would guess, the belief looks really good, but not your pnp results, can you visualize the raw points? I think your cuboid order is off @nv-jeff I think this is similar to the error the other person is having.

[brumocas]

I am not familiar with the raw points, can you elaborate on how to get them?
However, I think you are referring to the points attributed to each cuboid vertex.

Training Dataset orientation
I have checked my training data and the debug looks like this:

Based on this generated data the X indicating the box orientation is in the Nutrition Facts side of the cracker 3D model

Config file
After that I edited my config file and saw that I had different results when using the training vs the paper weight.
In the end the config stayed like this:

# Cuboid dimension in cm x,y,z
dimensions: {
    # For my training
    "cracker": [7.179999828338623, 16.403600692749023, 21.343700408935547],
    # For paper weights
    #"cracker": [16.403600692749023, 21.343700408935547,7.179999828338623] 
}

I had to follow the orientation that my 3D object had in meshlab as you can see in my first meshlab screenshot available in this issue (top left corner [size is in meters]) . You guys probably had a different one in your 3D model.

Inference results
Some inference new inference results:

I think that the output now is closer to your paper training weight, but yours is something else that I am not capable to achieve :(
Any suggestion in how to improve my results? How did you build your dataset?

Future Work
If I want to apply this to train a novel 3D object do you think that Diff-DOPE might be a great option to obtain the final pose?

[TontonTremblay]

can you draw these https://github.com/NVlabs/Deep_Object_Pose/blob/master/ros1/dope/src/dope/inference/cuboid_pnp_solver.py#L101

obj_2d_points

so I think when you call obj_2d_points and obj_3d_points; the points are not aligned, for example let say the top right front is at index 0 in obj_2d_points but index 1 in obj_3d_points, that will cause your issues.

possibly might be caused by not using the right of these 2. Which now I do not remember which is which. https://github.com/NVlabs/Deep_Object_Pose/blob/master/ros1/dope/src/dope/inference/cuboid.py#L82

@nv-jeff Maybe we should clear all of this so there is only one way to generate a set of cuboid points.

[brumocas]

Hello again, and sorry for the delayed response.
I managed to solve the problem by changing the cuboid_pnp_solver.py and cuboid.py with the ones you suggested, only copy and paste.

Inference results
This is how my inference looks:
00000.json

00100.json

00200.json

00300.json

00400.json

00500.json

00600.json

00700.json

00800.json

00900.json

01000.json

01100.json

01200.json

01300.json

01400.json

01500.json

01600.json

01700.json

01800.json

01900.json

02000.json

02100.json

I think this is it for now, I will continue to play with DOPE
Thanks 💪

[TontonTremblay]

ahhh this looks amazing good job, sorry I will check with @nv-jeff to see if he could make the update.

[nv-jeff]

Good sleuthing! I will take a look at the code and refactor the changes mentioned above.