About training time and iteration number

[ZJWang9928]

Hi, it's 18K × 32 samples.

Originally posted by @markkua in #150

Thank you for your quick reply.

I’d like to understand whether the model is expected to converge after processing the full 18K × 32 samples. If so, is it feasible to complete this within 2.5 days?

For reference, I trained Marigold on a single A800 GPU, and after 2.5 days it only completed about 2.3K iterations, with each iteration taking roughly 90 seconds.

[nandometzger]

This sounds like a dataloading issue rather than a GPU issue. I recommend testing the dataloader separatly to isolate the problem.

[yiyulics]

Hi, @ZJWang9928. I had the same issue. It takes about 70 seconds to process one iteration on a single A100 GPU. Theoretically, it will take 70 * 18,000 / 60 / 60 / 24 = 14.5 days.

Do you have any clues about the dataloader?

[Night1099]

Having same problem, iterations are very slow and Marigold 2 does not fit on a 4090, dataloader doesnt do any augmentation and is still slow, dataset is raw files on an nvme

For any other lost souls, the only way this has gotten down to a reasonable training time is using torch 2.8.0 and a 5090, the 32 effective bath size is using 98% of 32 gb so no idea how they managed to get it on 24GB and this train will still be 5 days

[markkua]

Hi @Night1099 , I would suggest checking the IO speed of the dataset first.

About the memory usage, torch is greedy in memory management and will sometimes occupy more memory than necessary especially on larger gpus. On the other hand, please check if you are using the same resolution, 480x640, as we did for training (see here)

[Night1099]

ah i see yes once data set is pre sized to 480x640 a 4090 works 🐙 thanks

[yiyulics]

Hi @Night1099 , I would suggest checking the IO speed of the dataset first.

About the memory usage, torch is greedy in memory management and will sometimes occupy more memory than necessary especially on larger gpus. On the other hand, please check if you are using the same resolution, 480x640, as we did for training (see here)

Hi @markkua, I've discovered an issue. In the __getitem__ function, the if statement DatasetMode.TRAIN == self.mode always yields to False during debugging (even if self.mode is set to DatasetMode.TRAIN). Consequently, the training process in the release version does not use preprocessing methods like resizing or augmentation, which leads to large resolution training and cannot fit into one GPU.

Marigold/src/dataset/base_normals_dataset.py

Lines 93 to 100 in b1dffaa

	def __getitem__(self, index):
	rasters, other = self._get_data_item(index)
	if DatasetMode.TRAIN == self.mode:
	rasters = self._training_preprocess(rasters)
	# merge
	outputs = rasters
	outputs.update(other)
	return outputs

The issue arises because the left-hand side Enum DatasetMode is defined in the same file, whereas the DatasetMode in the right-hand side self.mode is imported from src/dataset/base_depth_dataset.py, making them different classes and not equal.

A workaround would be:

if DatasetMode.TRAIN.value == self.mode.value: