Feat: Progressive sequence-length Curriculum Learning in the Dataloader

[aniruddhaadak80]

Problem

The current data loader in prepare.py strictly packs documents to MAX_SEQ_LEN (2048). Training on full context lengths from step 0 is often computationally wasteful and slows down early convergence.

Proposal

Modify the dataloader to support sequence-length warmup (Curriculum Learning). For example, start training with a sequence length of 256 for the first 10% of the training budget, and progressively double it until reaching 2048. This requires updating the make_dataloader logic to dynamically resize the sequence packing on the fly.

[MustakimShaikh01]

This seems like a great optimization. Starting with shorter contexts could speed up early training and reduce unnecessary compute before the model learns basic token dependencies.

One possible implementation could be a curriculum schedule such as:

• 0–10% steps → seq_len = 256
• 10–30% → seq_len = 512
• 30–60% → seq_len = 1024
• 60–100% → seq_len = 2048

The dataloader packing function could reference a scheduler variable that dynamically updates MAX_SEQ_LEN during training instead of fixing it in prepare.py.

Curious if this would require re-packing the dataset or if dynamic padding/truncation would be sufficient.