Replication Problems #23
Description
Hi,
I'm currently trying to replicate the results that you obtained on the CANOPUS dataset (8.13% Top--1 Accuracy) using the DiffMS Canopus checkpoint. However, I am consistently getting 0 on all metrics, as shown below:
test/acc_at_1 0.0
test/acc_at_10 0.0
test/acc_at_2 0.0
test/acc_at_3 0.0
test/acc_at_4 0.0
test/acc_at_5 0.0
test/acc_at_6 0.0
test/acc_at_7 0.0
test/acc_at_8 0.0
test/acc_at_9 0.0
test/cosine_at_1 0.0
test/cosine_at_10 0.0
test/cosine_at_2 0.0
test/cosine_at_3 0.0
test/cosine_at_4 0.0
test/cosine_at_5 0.0
test/cosine_at_6 0.0
test/cosine_at_7 0.0
test/cosine_at_8 0.0
test/cosine_at_9 0.0
test/tanimoto_at_1 0.0
test/tanimoto_at_10 0.0
test/tanimoto_at_2 0.0
test/tanimoto_at_3 0.0
test/tanimoto_at_4 0.0
test/tanimoto_at_5 0.0
test/tanimoto_at_6 0.0
test/tanimoto_at_7 0.0
test/tanimoto_at_8 0.0
test/tanimoto_at_9 0.0
test/validity 0.0
I am running DiffMS with the following parameters:
python src/spec2mol_main.py general.name=canopus_test dataset=canopus general.load_weights=checkpoints/diffms_canopus_pl_.ckpt train.eval_batch_size=4096 general.test_only=checkpoints/diffms_canopus_pl.ckpt
What I already verified:
- Checkpoints are loaded correctly, and the weights of the model correspond to the one in the checkpoint
- Metrics are working correctly. The molecules are just nonsensical.
Any assistance in replicating the results would be greatly appreciated.
I am using a fresh conda environment with torch 2.3.1 as suggested in the ReadMe. The installed packages are listed below:
Package Version Editable project location
------------------------ ----------- ------------------------------------------------------------------
aiohappyeyeballs 2.6.1
aiohttp 3.13.2
aiosignal 1.4.0
annotated-types 0.7.0
antlr4-python3-runtime 4.9.3
asttokens 3.0.1
async-timeout 5.0.1
attrs 25.4.0
certifi 2025.8.3
chardet 5.2.0
charset-normalizer 3.4.4
click 8.1.8
contourpy 1.3.0
cycler 0.12.1
decorator 5.2.1
diffms 1.0.0 analytical_v2/mixtures/msms/DiffMS
eval_type_backport 0.3.0
exceptiongroup 1.3.0
executing 2.2.1
filelock 3.19.1
fonttools 4.59.1
freetype-py 2.3.0
frozenlist 1.8.0
fsspec 2025.10.0
gitdb 4.0.12
GitPython 3.1.45
greenlet 3.2.4
h5py 3.14.0
hydra-core 1.3.2
idna 3.11
importlib_resources 6.5.2
ipdb 0.13.13
ipython 8.18.1
jedi 0.19.2
Jinja2 3.1.6
joblib 1.5.2
kiwisolver 1.4.7
lightning-utilities 0.15.2
MarkupSafe 3.0.3
matplotlib 3.7.1
matplotlib-inline 0.2.1
mpmath 1.3.0
multidict 6.7.0
munkres 1.1.4
myopic_mces 1.0.1
networkx 3.2.1
numpy 1.23.0
nvidia-cublas-cu12 12.1.3.1
nvidia-cuda-cupti-cu12 12.1.105
nvidia-cuda-nvrtc-cu12 12.1.105
nvidia-cuda-runtime-cu12 12.1.105
nvidia-cudnn-cu12 8.9.2.26
nvidia-cufft-cu12 11.0.2.54
nvidia-curand-cu12 10.3.2.106
nvidia-cusolver-cu12 11.4.5.107
nvidia-cusparse-cu12 12.1.0.106
nvidia-nccl-cu12 2.20.5
nvidia-nvjitlink-cu12 12.9.86
nvidia-nvtx-cu12 12.1.105
omegaconf 2.3.0
overrides 7.3.1
packaging 25.0
pandas 1.4.0
parso 0.8.5
pexpect 4.9.0
pillow 11.3.0
pip 25.2
platformdirs 4.4.0
prompt_toolkit 3.0.52
propcache 0.4.1
protobuf 6.33.1
psutil 7.1.3
ptyprocess 0.7.0
PuLP 3.3.0
pure_eval 0.2.3
pycairo 1.28.0
pydantic 2.12.4
pydantic_core 2.41.5
Pygments 2.19.2
pyparsing 3.2.3
python-dateutil 2.9.0.post0
pytorch-lightning 2.0.4
pytz 2025.2
PyYAML 6.0.3
rdkit 2024.9.4
reportlab 4.4.1
requests 2.32.5
rlPyCairo 0.2.0
scikit-learn 1.6.1
scipy 1.13.1
seaborn 0.13.2
sentry-sdk 2.44.0
setuptools 68.0.0
six 1.17.0
smmap 5.0.2
SQLAlchemy 2.0.43
stack-data 0.6.3
sympy 1.14.0
threadpoolctl 3.6.0
tomli 2.3.0
torch 2.3.1
torch_geometric 2.3.1
torchmetrics 0.11.4
tqdm 4.67.1
tqdm_joblib 0.0.4
traitlets 5.14.3
triton 2.3.1
typing_extensions 4.14.1
typing-inspection 0.4.2
tzdata 2025.2
unicodedata2 16.0.0
urllib3 2.5.0
wandb 0.23.0
wcwidth 0.2.14
wheel 0.45.1
yarl 1.22.0
zipp 3.23.0```