How to resume the training process from specific checkpoint?

I tried the training several times and one time, my training process was broken. So, it generated several folders in output_dir. How can I resume the training process from specific checkpoint in these folders?