Questions about AR -> SDAR

[SSSSSSuger]

Thank you for your great work on this project!

I noticed that you have open-sourced the SFT code.
May I ask whether it's possible to use this code—by modifying the perturbation logic as described in your statement,
"During pretraining, the entire sequence is perturbed, whereas during SFT, the prompt remains clean and the response is perturbed"—to carry out the continued pre-training mentioned in Section 5.1.1 of your paper?

Thanks again for your time and for sharing your work openly!

[chengshuang18]

Thanks for your interest and for the kind words!

Yes — in principle you can use the released SFT code to run the continued pre-training in Sec. 5.1.1, but it doesn’t require complicated changes to the perturbation logic.

The key difference is the loss mask:

• In pretraining / continued pretraining, the entire sample should contribute to the loss (i.e., prompt + response are all labels, not masked). If you do this, the model will automatically learn with the prompt portion perturbed as well, since those tokens are no longer excluded from the loss.
• In SFT, we keep the prompt clean and perturb only the response mainly because the prompt tokens are typically masked out from the loss (labels set to -100), so perturbing them would not be trained against.

Concretely, in the current LLaMAFactory SFT pipeline, the dataset preprocessing usually sets prompt labels to -100, meaning the prompt does not participate in training loss. So for continued pretraining, you have two options:

1. Simpler: implement a custom dataset preprocessing that treats the whole sequence as labels (no -100 for the prompt).
2. More elegant: add an “SDAR-style” preprocessing option under LLaMAFactory’s pretraining setting, so you can reuse the pretraining data flow while keeping the SDAR perturbation.

If there’s demand, we can consider adding an official continued-pretraining preprocessing path in a future update (no fixed schedule yet).

For easier discussion, you’re also welcome to join our WeChat group — the QR code is available in the README.

[SSSSSSuger]

Your two proposed implementation approaches are both very helpful. I’ll follow your suggestions and try implementing a customized preprocessing pipeline. If an official continued-pretraining workflow becomes available in the future, I’d be excited to try it as well.

Thank you again for your patient and detailed response!

Thanks for your interest and for the kind words!

Yes — in principle you can use the released SFT code to run the continued pre-training in Sec. 5.1.1, but it doesn’t require complicated changes to the perturbation logic.

The key difference is the loss mask:

• In pretraining / continued pretraining, the entire sample should contribute to the loss (i.e., prompt + response are all labels, not masked). If you do this, the model will automatically learn with the prompt portion perturbed as well, since those tokens are no longer excluded from the loss.
• In SFT, we keep the prompt clean and perturb only the response mainly because the prompt tokens are typically masked out from the loss (labels set to -100), so perturbing them would not be trained against.

Concretely, in the current LLaMAFactory SFT pipeline, the dataset preprocessing usually sets prompt labels to -100, meaning the prompt does not participate in training loss. So for continued pretraining, you have two options:

1. Simpler: implement a custom dataset preprocessing that treats the whole sequence as labels (no -100 for the prompt).
2. More elegant: add an “SDAR-style” preprocessing option under LLaMAFactory’s pretraining setting, so you can reuse the pretraining data flow while keeping the SDAR perturbation.

If there’s demand, we can consider adding an official continued-pretraining preprocessing path in a future update (no fixed schedule yet).

For easier discussion, you’re also welcome to join our WeChat group — the QR code is available in the README.