After using SimPO, the model generation results contain many loops

[ZetangForward]

Hi, here are my training settings:

• I used SimPO on the LLama3.1-8B-Instruct model with the recommended settings of Llama3-8B-Instruct-V2: gamma=10, gamma_beta_ratio=0.3, lr=1e-6
• I trained the model on the Instruction dataset (Long-Alpaca)
• I randomly provided the model with a prompt during the inference, the ground truth should be: Garden

Here is the inference result of the vanilla LLama3.1-8B-Instruct model: The milk is in the garden.

However, after fine-tuning with SimPO loss, the reference results seem like this:

"pred": " \n\nThe football is in the hallway.  The apple is in the hallway.  The apple is in the garden. The apple is in the hallway. The football is in the hallway. The apple is in the garden.  The apple is in the hallway. The football is in the kitchen. The football is in the hallway. The apple is in the garden. The apple is in the hallway. The football is in the hallway. The football is in the kitchen. The football is in the"}

It contains heavy loops.

Based on your experience, how can I modify my hyperparameters to avoid this situation?

[yumeng5]

Hi,

It's generally recommended to tune hyperparameters on different datasets, and we found the learning rate to be the most important one to tune. If you are seeing heavy repetitions, you may try to reduce the learning rate. gamma and gamma_beta_ratio can usually be set to the default values (though you might want to further tune them for optimal results).

Best,
Yu