Performance Degradation in HumanML3D

[yina427]

Hello,
▪️I followed the instructions on GitHub step by step to prepare the HumanML3D dataset and conducted an evaluation.

▪️I only performed the evaluation using the humanml-encoder-512 pretrained model, yet I observed a performance degradation.

▪️It seems like there is an issue with the dataset or the process, but I am unable to determine the cause, so I would like to ask for help.

For reference, I followed all the instructions for HUMANML3D, and the number of files in motion-diffusion-model/dataset/HumanML3D/new_joint_vecs was 29,228.

Best regards

[GuyTevet]

I saw this issue with the older model regardless of the data, I would like to suggest two thing to work around until we find the issue:

(1) Evaluate the 50-steps model 50 diffusion steps model
(2) Revert to commit 63edacf and evaluate from there

Please let me know if that was helpful.
I will mark this as a bug, but until we find it, let's provide other people with feedback on how to avoid it.

[yina427]

Thanks for the guidance.
I followed the instructions, but the result is still the same.

(P.S. The last error seems to be related to the next issue.)
#235 (comment)

[GuyTevet]

Let’s try to understand if that is because of the data. Please try to use the same data I am using:

https://drive.google.com/drive/folders/1OZrTlAGRvLjXhXwnRiOC-oxYry1vf-Uu?usp=drive_link

[GuyTevet]

To resolve the numpy issue just downgrade it to the version specified in the requirements.

[yina427]

Hello,
I have reinstalled the environment and redownloaded the dataset from the official source,
except for texts.zip, which I downloaded from the HumanML3D GitHub.

However, there is no significant difference in performance.

I would really appreciate it if you could suggest any further steps I could try.

[GuyTevet]

It seems that you got very similar results to the official table (FID -s now .57 instead of the 3.1 you got last time).

[yina427]

Oh, thank you.
Thanks to the data you provided, there is a significant improvement in the FID score.
However, the R-precision is still lower compared to the baseline log.

[ BASE LINE log ]

[ MY log ]

However, evaluate_matching_score, unlike evaluate_fid, involves text, so I suspect the problem lies in text_embeddings itself or the distance calculation.

(For reference, the data you provided didn't include texts.zip, so I downloaded it from HumanML3D's GitHub: HumanML3D/texts.zip.)

I'll keep debugging, and if I find a solution, I'll share it.

[GuyTevet]

It is not possible to completely reproduce my results, because of the stochasticity of the evaluation process.
The delta of the R-precision is smaller (in the error range to my understanding).
Anyway, that's the reason we run 20 repetitions, to reduce the error range. Please check the mean results at the end of the log, instead of the results of repetition 0.

[yina427]

Thank you.
The generated results after multiple repetitions are as follows:

[ BASE LINE log ]

[ MY log ]

As I understand, the result report can be summarized as:

• The R-precisions for both Real and MDM-50steps are lower than the error range.

• The FID score is lower for MDM-50steps.

Please let me know if I misunderstood anything.
It doesn't seem to be a data issue, so I will take some time to think it over.

[yina427]

Hello
This log is the result of someone else doing it from scratch in a different environment.
Likewise, it seems to be worse than the overall error range than the performance table.

Thank you.