SequencerCriterion assumes that the time step is on the first index (on line 41 for example - https://github.com/Element-Research/rnn/blob/master/SequencerCriterion.lua#L41) So I believe it will do wonky things in batch mode.