cifia 的center loss 的训练过程不收敛

[JinmingZhao]

您好，
下面是几个阶段的训练输出，center-loss 下降的特别快，而softmax loss 基本不动，随着训练进行，centerloss 逐渐增加，softmax loss 逐渐下降，在其他的数据集上训练过程也是如此，这样正常吗，能否解释一下这个过程的原因？ (正常情况下 l2 loss 也是这样，一般先降 l2 loss 然后再降 softmax loss， 这样训练就特别慢, 您能帮忙解释一下不？) 。

谢谢！
step: 0, training accuracy: 0.01, training loss: 9.13, center_loss_value: 8.95, softmax_loss_value: 4.60
8.9476
step: 100, training accuracy: 0.00, training loss: 4.60, center_loss_value: 0.00, softmax_loss_value: 4.60
0.000475059
step: 200, training accuracy: 0.00, training loss: 4.60, center_loss_value: 0.00, softmax_loss_value: 4.60
0.000334069
step: 300, training accuracy: 0.00, training loss: 4.61, center_loss_value: 0.00, softmax_loss_value: 4.61
0.000232262
...
step: 7000, validation accuracy: 0.00, validation loss: 4.61
step: 7000, training accuracy: 0.02, training loss: 4.61, center_loss_value: 0.01, softmax_loss_value: 4.60
0.00856465
step: 7100, training accuracy: 0.00, training loss: 4.60, center_loss_value: 0.01, softmax_loss_value: 4.60
0.00611517
step: 7200, training accuracy: 0.03, training loss: 4.59, center_loss_value: 0.01, softmax_loss_value: 4.59
0.00610098
step: 7300, training accuracy: 0.02, training loss: 4.60, center_loss_value: 0.01, softmax_loss_value: 4.59
0.00591285
...
step: 10000, validation accuracy: 0.02, validation loss: 4.60
step: 10000, training accuracy: 0.03, training loss: 4.60, center_loss_value: 0.02, softmax_loss_value: 4.55
0.0197095
step: 10100, training accuracy: 0.05, training loss: 4.58, center_loss_value: 0.02, softmax_loss_value: 4.56
0.019367
step: 10200, training accuracy: 0.01, training loss: 4.59, center_loss_value: 0.02, softmax_loss_value: 4.56
0.0246912
step: 10300, training accuracy: 0.00, training loss: 4.59, center_loss_value: 0.03, softmax_loss_value: 4.56
0.0319362
step: 10400, training accuracy: 0.03, training loss: 4.56, center_loss_value: 0.03, softmax_loss_value: 4.53
0.0250489
....
step: 15000, validation accuracy: 0.03, validation loss: 4.59
step: 15000, training accuracy: 0.02, training loss: 4.59, center_loss_value: 0.06, softmax_loss_value: 4.48
0.0584633
step: 15100, training accuracy: 0.01, training loss: 4.55, center_loss_value: 0.07, softmax_loss_value: 4.49
0.0664418
step: 15200, training accuracy: 0.04, training loss: 4.48, center_loss_value: 0.05, softmax_loss_value: 4.43
0.0492492
step: 15300, training accuracy: 0.05, training loss: 4.53, center_loss_value: 0.05, softmax_loss_value: 4.48
...step: 79000, validation accuracy: 0.09, validation loss: 12.37
step: 79000, training accuracy: 0.25, training loss: 12.37, center_loss_value: 0.12, softmax_loss_value: 2.98
0.117979
step: 79100, training accuracy: 0.23, training loss: 3.09, center_loss_value: 0.11, softmax_loss_value: 2.98
0.11036
step: 79200, training accuracy: 0.25, training loss: 3.10, center_loss_value: 0.12, softmax_loss_value: 2.98
0.118568
step: 79300, training accuracy: 0.23, training loss: 3.15, center_loss_value: 0.16, softmax_loss_value: 2.99
0.161232
step: 79400, training accuracy: 0.23, training loss: 3.24, center_loss_value: 0.22, softmax_loss_value: 3.01
0.220702
step: 79500, training accuracy: 0.21, training loss: 3.14, center_loss_value: 0.16, softmax_loss_value: 2.98
0.159162
step: 79600, training accuracy: 0.21, training loss: 3.09, center_loss_value: 0.12, softmax_loss_value: 2.97
0.121266
step: 79700, training accuracy: 0.22, training loss: 3.18, center_loss_value: 0.18, softmax_loss_value: 2.99
0.184051
step: 79800, training accuracy: 0.18, training loss: 3.14, center_loss_value: 0.17, softmax_loss_value: 2.98
0.1673
step: 79900, training accuracy: 0.19, training loss: 3.13, center_loss_value: 0.15, softmax_loss_value: 2.98

[UpCoder]

您好，我仅说说我自己的观点。
我个人认为因为center loss的降低是比较容易的，假设最极端的情况下，也就是无监督，我们不优化softmax loss。在这种情况下，center loss很容易变成0。这是为什么呢？
因为center loss只是计算的学习的特征与中心之间的差距，而且学习得到的特征和center features都是在反向传播中不断更新的。所以，我们只需要让学习得到的特征和center features一致，也就是说不管输入是什么，输出都是一样的。那么对应的center loss就一定是0。对于我们的网络来说，要实现上述功能很简单，甚至一层就可以实现。但明显上述网路的输出是错误的，并不能实现我们的分类功能。

当有softmax loss的时候，因为我们是寻找局部最优点，也许center loss对应的局部最优点很容易找到，但是那个点并不是softmax loss的最优点，所以一开始找到了center loss的局部最优点，但是在后续训练的过程中，会不断的寻找整体的局部最优点，在center loss对应的局部最优点和softmax loss所对应的局部最优点之间寻找一个平衡。

至于l2 loss应该是和center loss一样的道理。

上面的只是我个人的想法，并没有严密的数学逻辑推导证明，如果您有什么更好的解释方法，也希望咱们可以多多沟通交流！

[JinmingZhao]

这样说比较直观，比较好理解，非常感谢！ 另外一个问题就是 lambda的值有没有好的建议？ 一般l2的话我们取10-4左右，这里如果取0.1的话跟 softmax 基本是一个量级，但是这样的话，center loss 迅速的降低，也符合你的理解，太小，比如10-4, 的话也会降的很快，但是如果lambda取0.003（作者论文中结论） center loss 基本将到0.02左右，不会降到太小，然后就能和 softmax 就会基本同步下降。 这样lambda值的选取您有没有好的见解？（同l2 loss 的weight decay 系数）

[UpCoder]

我感觉lambda是一个hyper-parameter，应该是要通过实验结果来调整的，一般的话，就10-1~10-4试吧。不过实验结果给我的感觉是不同大小的lambda对分类的准确率是没有提升的。lambda越大的话，最后收敛的就越慢，当然不同类别的center features之间的距离也会变大。

[wangjue-wzq]

@JinmingZhao @UpCoder 大佬好，我这样做和仅使用softmax没有性能的提升，（lambda，我尝试过1，0.1，0.01）不知为何:
`

def center_loss(labels,features, alpha,lambda_c,lambda_g,num_classes):

len_features = features.get_shape()[1]
with tf.variable_scope('v_center',reuse = True):
      centers = tf.get_variable('centers', [num_classes, len_features], dtype=tf.float32,
                          initializer=tf.constant_initializer(0), trainable=False)
labels = tf.argmax(labels, axis=1)
centers_batch = tf.gather(centers, labels)
center_loss = tf.reduce_mean(tf.square(features - centers_batch))
centers_update_op = tf.scatter_sub(centers, labels, diff) 
with tf.control_dependencies([centers_update_op]):
    combo_loss = lambda_c * center_loss + softmax_loss(labels,features)
return combo_loss

total_loss = lossclass.center_loss(alpha=0.5,lambda_c=0.1,num_classes=num_classes)
custom_vgg_model.compile(loss=total_loss,optimizer= sgd,metrics=['accuracy'])

`

[UpCoder]

centers_update_op是否有更新？是在mnist数据集上做的吗？