training getting slow after loading model ckpts

[10wook]

Hi ... its me again.

Hope you miss me haha,,

unfortunately I ma here with some issues.

I have finally succeeded to train your models in eng_kor version.

and I found out that training datasets of 100,000 pics with 700,000 iter was not enough to make some performance that I want.

SO i decided to train more with same dataset.

and then i found out that s/iter has got bigger

before loading the models it costed like 2.46 sec per iter and now it takes at least 4.85 sec per iter.

It usually takes more than 20 sec per iters now.

I found out that my gpus arent working.

han@han-System-Product-Name:~$ nvidia-smi
Tue Apr 22 02:04:17 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.135 Driver Version: 550.135 CUDA Version: 12.4 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 4090 Off | 00000000:01:00.0 On | Off |
| 35% 42C P8 24W / 450W | 22360MiB / 24564MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 1707 G /usr/lib/xorg/Xorg 552MiB |
| 0 N/A N/A 1892 G /opt/teamviewer/tv_bin/TeamViewer 20MiB |
| 0 N/A N/A 1958 G /usr/bin/gnome-shell 90MiB |
| 0 N/A N/A 4783 G ...seed-version=20250417-180112.233000 43MiB |
| 0 N/A N/A 37012 G ...erProcess --variations-seed-version 249MiB |
| 0 N/A N/A 543720 C python 21370MiB |
+-----------------------------------------------------------------------------------------+

I really want to know why this is happening.

Any ideas?? plz help me out

[10wook]

Hi, I decided to give up last night. I just made my move to just wait even though the training speed was too slow.
Then I had a deep rest sleeping and when I woke up I realized that training speed has got faster!
I came back up to 2.5s per iter.
I cannot tell the reason why it was slow at the first time but i can tell patient might be an answer.
Still I am interested in the reason so if I find any answer I will make a comment.

[Khurdhula-Harshavardhan]

+1

[10wook]

+1
@Khurdhula-Harshavardhan
Do you have any answers? Or you just have the same problem?

[prajwalgatti]

If the code, data type, and batch size are the same as before, there could be factors outside of the code that might be slowing down the training process. For example, if there are other processes running on the machine with heavy CPU, memory, or IO load, it could slow down your training process. You can try to stop them to improve your iterations/sec. Also, sometimes it could just be that your GPU device is overheated.

Beyond these reasons, if the code is still slow, try profiling to find bottlenecks. If dataloading is the bottleneck, I would suggest exploring some tricks to optimize dataloading speed. Try playing with prefetch_factor (link), num_workers (link), and pin_memory (link) in your PyTorch data loader code.