How to reduce training loss with a limited dataset?

[Dr-xinyu]

I am currently training a dual-arm gripper(2 robotiq_2f_85) model using a relatively small custom dataset. The dataset consists of over 400 objects, with 2,000 generated grasp poses per object。 I modified the PickDataset to create a dualarmdataset and adjusted the training shell script from the tutorial to run on two GPUs. I just modified the input data dimensions passed to the network, leaving the rest of the architecture unchanged
However, I am encountering convergence issues:

Generator Loss: Stagnates around 2.5 after approximately 1k epochs and does not decrease further, even up to 4k epochs.

Discriminator Loss: The total loss across both GPUs stabilizes around 1.3 after 1k epochs and remains flat.

Output Quality: The generated grasp poses are completely unusable.

Aside from expanding the dataset (which is time-consuming), are there any other strategies or specific hyperparameters I should adjust to improve convergence?
my generator's sh script

my discriminator's script I would greatly appreciate your help！

[adithyamurali]

Hi @Dr-xinyu

Great that you have modified the training code for a dual-arm setup. Instead of the loss, could you check the following metrics listed in the logs? These metrics are a better indication on the training process convergence.

Additionally, I would suggest to finetune the network rather than train from scratch (since your dataset is still sizable but still small). You can pass in a pretrained checkpoint as input

Generator:
For best perf, the validation the grasp reconstruction error reconstruction/error_trans_l2 should be a few cm

Discriminator:
For best perf, Validation AP score should be > 0.8

Taken from:
https://github.com/NVlabs/GraspGen/blob/main/tutorials/TUTORIAL.md#monitoring-training
https://github.com/NVlabs/GraspGen/blob/main/tutorials/TUTORIAL.md#monitoring-training-1