Data missing in Zenodo v2 archive, RMSD calculation errors, and request for pre-computed metrics

[2023ComputBio]

Hi @CAODH ,

First of all, thank you for your excellent work on MolGenBench and for making the code and data publicly available. It’s a very valuable benchmark for the community.

I’ve been trying to reproduce the evaluation results using the version2 data uploaded to Zenodo and have encountered a few issues that I wanted to bring to your attention.

1. Missing files in Zenodo dataset

When running the provided evaluation scripts, several .sdf files from generative sampling or vina docking appear to be missing or empty, which prevents the full reproduction of the reported results for certain methods/targets. Below are some of the missing files I’ve identified:
TargetDiff:
P06239/Round2/De_novo_Results/TargetDiff_generated_molecules/P06239_TargetDiff_generated_molecules_vina_docked.sdf is missing.
P06239/Round3/De_novo_Results/TargetDiff_generated_molecules/P06239_TargetDiff_generated_molecules_vina_docked.sdf is missing.
DecompDiff:
P00915/Round1/De_novo_Results/DecompDiff_generated_molecules/P00915_DecompDiff_generated_molecules.sdf is missing.
P00915/Round1/De_novo_Results/DecompDiff_generated_molecules/P00915_DecompDiff_generated_molecules_vina_docked.sdf is missing.
...
ShapeMol_Hit_to_Lead:
P55055/Round1/Hit_to_Lead_Results/Sries63387/ShapeMol_Hit_to_Lead/P55055_Sries63387_ShapeMol_Hit_to_Lead.sdf is missing.
P55055/Round1/Hit_to_Lead_Results/Sries63387/ShapeMol_Hit_to_Lead/P55055_Sries63387_ShapeMol_Hit_to_Lead_vina_docked.sdf is missing.
...
shepherd_x1x3x4_mosesaq_submission_Hit_to_Lead:
Q07817/Round1/Hit_to_Lead_Results/Sries44140/shepherd_x1x3x4_mosesaq_submission_Hit_to_Lead/Q07817_Sries44140_shepherd_x1x3x4_mosesaq_submission_Hit_to_Lead.sdf is missing.
Q07817/Round1/Hit_to_Lead_Results/Sries44140/shepherd_x1x3x4_mosesaq_submission_Hit_to_Lead/Q07817_Sries44140_shepherd_x1x3x4_mosesaq_submission_Hit_to_Lead_vina_docked.sdf is missing.
...
DeleteHit2Lead(CrossDock)_Hit_to_Lead:
P42336/Round1/Hit_to_Lead_Results/Sries23245/DeleteHit2Lead(CrossDock)_Hit_to_Lead/P42336_Sries23245_DeleteHit2Lead(CrossDock)_Hit_to_Lead.sdf is missing.
P42336/Round1/Hit_to_Lead_Results/Sries23245/DeleteHit2Lead(CrossDock)_Hit_to_Lead/P42336_Sries23245_DeleteHit2Lead(CrossDock)_Hit_to_Lead_vina_docked.sdf is missing.
...
DiffDec_Hit_to_Lead:
Q05397/Round1/Hit_to_Lead_Results/Sries59918/DiffDec_Hit_to_Lead/Q05397_Sries59918_DiffDec_Hit_to_Lead.sdf is missing.
Q05397/Round1/Hit_to_Lead_Results/Sries59918/DiffDec_Hit_to_Lead/Q05397_Sries59918_DiffDec_Hit_to_Lead_vina_docked.sdf is missing.
Q05397/Round2/Hit_to_Lead_Results/Sries59918/DiffDec_Hit_to_Lead/Q05397_Sries59918_DiffDec_Hit_to_Lead.sdf is a empty file.
...

This causes the evaluation pipeline to skip these entries or raise Error (OSError: File error: Invalid input file). Could you please check and, if possible, re-upload the complete dataset?

2. RMSD calculation failures

During the RMSD metric calculation, many molecules fail with the following error:
[RMSDMetric] Symmetry RMSD failed: Graphs are not isomorphic. Make sure graphs have the same connectivity.

My hypothesis is that this might be caused by a mismatch in molecular topology (connectivity) between the original generated molecules and the Vina-docked output (possibly during the .pdbqt to .sdf conversion). Has this issue been encountered before? If there is a known fix or preprocessing step to ensure graph isomorphism, it would be very helpful to know.

3. Request for pre-computed evaluation metrics

Currently, the Zenodo archive contains the generated molecules and docked outputs, but to obtain the final evaluation metrics (e.g., hit rediscovery rate, RMSD, IFP score), users must run the entire evaluation pipeline locally. This is quite time-consuming, and the provided Jupyter notebooks for visualization are somewhat difficult to follow.

Would it be possible for you to release the pre-computed evaluation results (e.g., CSV/JSON files with all calculated metrics) alongside the molecular data? This would greatly facilitate reproducibility, analysis, and comparison for users who wish to build upon your benchmark without re-running lengthy computations.

Additionally, if the notebooks could be refactored for better readability (e.g., with clearer sections, comments, and maybe a simplified plotting interface), that would be a great help for the community.

Thank you again for your hard work and for considering these points. I’m happy to provide more details or logs if needed.

[HShokaku]

Hi @2023ComputBio,

Thank you very much for your interest in MolGenBench and for carefully inspecting the v2 dataset. We really appreciate you taking the time to report these issues and help improve the benchmark.

Below we respond to your points one by one.

1. Missing files in Zenodo dataset

This behavior is expected in the current dataset and is not due to an accidental omission during upload.

• Missing or empty generated .sdf files indicate that the corresponding model failed to generate any valid molecules for that specific target / round. We have updated the code to skip empty .sdf files, preventing Invalid input file errors during evaluation.

• Missing Vina-docked .sdf files indicate that Vina docking failed for all generated molecules. In such cases, no docked poses were produced, so the docked file is absent.

2. RMSD calculation failures

These RMSD failures are caused by connectivity mismatches introduced during the .pdbqt → .sdf conversion. In the benchmark, we skip RMSD calculation for such cases. These cases are relatively rare compared to the total number of evaluated molecules.

3. Request for pre-computed evaluation metrics

We fully agree with your suggestions. We are currently preparing a release that will include pre-computed evaluation results. We will also refactor and simplify the visualization notebooks.

Thanks again for raising these points and helping us improve the benchmark! Please feel free to open follow-up issues if you encounter other problem.

[2023ComputBio]

Hi @HShokaku,

Thank you for your detailed response to my questions. I appreciate the clarifications regarding the missing files and RMSD calculation issues, as well as your efforts to address them in the updated code.

I would like to follow up on the point about handling generation failures in the evaluation. While I understand that skipping empty or invalid files during computation is a practical approach to avoid errors, I believe it might be worth considering how these cases are reflected in the final reported results.

If generation failures are entirely excluded from the evaluation, the reported metrics (e.g., success rates, docking scores, or other performance measures) could potentially overestimate the model's practical utility. For instance, as noted in your preprint, Delete and DiffDec demonstrate the top performance in the Hit2Lead scenario. However, in the current dataset, Delete failed to generate molecules in approximately 137 out of 600 cases, and DiffDec had a similar proportion of generation failures. If these failures are not accounted for in the final metrics, the results may not fully capture the models' robustness and reliability in real-world applications.

Would it be possible to treat "generation failure" as an explicit outcome in the evaluation framework? For example, by including such cases in the denominator when calculating success rates or by reporting failure rates alongside other metrics? This could provide a more comprehensive and realistic assessment of each model's performance.

Thank you again for your hard work and for considering these suggestions. I am eagerly looking forward to the pre-computed evaluation results and the refined visualization notebooks. Your contributions to the field are highly valued.

[kentoabeywardane]

Hello @HShokaku,

I'd like to echo 2023ComputBio's praise for this fantastic benchmark and requests for the precomputed evaluations for each model in the paper.

I also wanted to point out that when running the InteractionScoreMetric, I noticed that the prepared proteins for the following Uniprot ID's fail to be converted to prolif.Molecule objects: P21554, P00742, leading to the inability to compute this metric for the corresponding series.

• MolGenBench_Version2/P21554/P21554_prep.pdb
• MolGenBench_Version2/P00742/P00742_prep.pdb

I've tried both recommended approaches (by prolif's recommendations) of converting PDB files to prolif objects, but both seem to fail:

RDKit (current implementation):

protein = Chem.MolFromPDBFile(protein_path, removeHs=False)
plf_protein = plf.Molecule(protein)

MDAnalysis approach:

u = mda.Universe(protein_path, to_guess=['bonds'])
plf_protein = plf.Molecule.from_mda(u, "protein")

I set up the environment using the instructions in this repo, but to be explicit, I am using rdkit==2025.09.3, prolif==2.0.3, meeko==0.1.dev3, and mdanalysis==2.10.0.

Any suggestions would be helpful or if there is anything else I could provide, I'd be happy to do so.
Thank you!

[HShokaku]

Hi @kentoabeywardane,

Sorry for the late reply and thank you very much for your interest in MolGenBench!

Regarding the problem you encountered in InteractionScoreMetric, we were able to reproduce the same issue using the environment you provided. After further testing, we found that the following environment works well on our side: rdkit==2025.9.1, prolif==2.0.3, mdanalysis==2.7.0.

We have now updated the recommended environment configuration in the README accordingly. Thank you again for reporting this issue.

By the way, the precomputed evaluations are still under preparation. We will notify you as soon as they are ready!

Please feel free to let us know if you encounter any other issues or have further questions.
Thank you!

[HShokaku]

Hi @2023ComputBio , @kentoabeywardane

We have released Version 3 of the dataset. In this release, we added pre-computed evaluation CSV results to each results folder. We also uploaded the aggregated results under the paper_results folder in the repository.