Dropout rate should be set to 0 if not using dropout

[droid666]

Hi,
Thanks for this code.
From looking at it, I think the dropout rate should be set to 0 if not using dropout. Since it looks like the code is going to adapt the weights W based on that rate independent of whether dropout is used or not.

[yliu90]

I check the dropout code and find that issues bother me too the dropout flag seems to switch the way model calculate the gradient but I find that the theano graph of backpro mode is actually wrong ...

[mdenil]

Hi,

I don't understand the problem you are identifying. Can you explain in more detail?

The code always builds a theano graph for both dropout and no dropout regardless of whether dropout will be used or not during training. The dropout flag comes in on this line: https://github.com/mdenil/dropout/blob/master/mlp.py#L314 and chooses which part of the graph is used to compute updates.

[droid666]

Sorry missed your message.

The problem is with the case that you don't want to use dropout, but still have values given for dropout rates. Then the normal net will reduce the (initial) output per layer with the given dropout rates.

Because this is done on normal net:
W=next_dropout_layer.W * (1 - dropout_rates[layer_counter])

This is not a problem. It will learn fine. It is just not so intuitive. The intialization of W is messed up a little, if you chose them carefully but then multiply with whatever random value is still in dropout_rates.

I my local fork I removed that, so the normal net is a "pure net". Instead I do the adaption on the dropout net:
W=layer.W / (1 - dropout_rates[layer_counter])
But then layer.W must be used in params and not W.