The gap between train and inference

[youngsheen]

During training, the encoder only sees up to 30% of the tokens, but during inference, it needs to handle 0 to 100% of the tokens. Why doesn't this inconsistency between training and inference cause problems? (In LLMs, length (OOD) often leads to inference failure.)

[LTH14]

This is similar to what people observed in MAE. In images, seeing 30% of the tokens and predicting the others is a much harder task than seeing, say, 70% of the tokens and predicting the others. Together with the attention mechanism in transformer, this gives good generalization to unseen masking ratio.