Sequence Packing

[trolommonm]

1. Did you guys use sequence packing in your training?
2. If so, did you use special attention masks to "isolate each sample"? In other words, does tokens from one sample attend to tokens from another sample?

[RenShuhuai-Andy]

Hi,

1. yes, we use packing in training.
2. yes, we use flash_attn_varlen_func, which achieving "isolating each sample"