Loss always come to NaN after some epochs

[my wandb data board](https://wandb.ai/kaggle-gun/porn-labeling?workspace=user-kaggle-gun)
![image](https://github.com/curt-tigges/query2label/assets/52595463/78f45358-b4dd-4ce7-8a28-6269f9448da3)
![image](https://github.com/curt-tigges/query2label/assets/52595463/2147e79b-4098-4937-900f-1d38e97f3c07)
