Ranking speed & training hyperparameters #10
Description
I'm trying to replicate the Llama-3-8B setup with one of our custom finetunes and stumbled across some questions:
The ranking model gets called with a batch size of 1 and increasing that didn't seem to make the ranking any faster. With my current setup the ranking takes longer than the actual training. Is there a way to speed up the ranking part of the pipeline?
You mention in the paper that you train for 18 epochs per iteration, usually instruction tuning is done with a single epoch since the models can overfit on the data pretty quickly. Did you really end up training each iteration for 18 epochs and didn't that lead to massive overfitting?
Could you provide some loss numbers for iter 1/2/3 just so people have a number to compare their runs to? The loss seems very high but i'm not sure how the numbers are supposed to look like since SPPO uses a custom loss function.
Overall a pretty nice pipeline that you built with the iterative generation->ranking->training setup