Discriminator training in tutorial doesn't improve

[bkolligs]

Hello!
I'm trying to follow the tutorial for a single object and seeing the following results:
The generator trained well for 1000 epochs on 1 GPU, reaching a final error_trans_l2 value of 0.06695060431957245.
However, when I tried to train the discriminator, the metrics described in the tutorial that are supposed to improve did not:

Validation AP

# Step, metric, value
65, valid/metric/noise/ap, 0.5208368301391602
...
5265, valid/metric/noise/ap, 0.5173702836036682

BCE TopK Loss

It stays around this 0.69 value for the whole training run.

# Step, metric, value
1, train/loss/bce_topk, 0.7208134531974792
...
 6175, train/loss/bce_topk, 0.6932986378669739

Any thoughts on the issue here? I'm training on an A100 80GB.

[adithyamurali]

Hi @bkolligs

I will take a look at it this week, I'm not sure what the root cause is here. I've noticed that this happened sometimes at the beginning of training, but it would eventually go down normally when doing multi-gpu training. This is however becoming a issue as the tutorial is with a single gpu.

Could you try changing the backbone from "ptv3" to "pointnet". I've noticed that the latter is more stable. You just need to change one param: https://github.com/NVlabs/GraspGen/blob/main/tutorials/tutorial_train_dis.sh#L15

[bkolligs]

Thanks for the reply @adithyamurali! I changed the backbone to "pointnet" but am still seeing a stubborn loss value:

[2025-09-22 20:45:10,489][__main__][INFO] - Train Epoch 01  0010/0013  Data time 0.0995  Forward time 0.0482  Logging time 0.0075 Loss 0.6942
...
[2025-09-22 20:46:24,542][__main__][INFO] - Train Epoch 62  0010/0013  Data time 0.0692  Forward time 0.0440  Logging time 0.0085 Loss 0.6933

[adithyamurali]

I'm out this week, but I can take a look at this issue next week, so it'll have to wait.

• I would basically check the tensorboard logs to see if there are both positive and negative data in the batches. Having a imbalanced dataset would affect the training run.
• what if you run it longer?
• Does this problem still exist with multi-gpu training?
• I would set this param to 1.0 (basically only train on mesh points, a less complex training problem than training with partial point clouds) and try again. I would delete any debug folders before restarting training

[mingqing431]

i meet the same issue, could you give me some advice, i already set this param to 1.0 with 1 GPU training