Missing diffusion model checkpoint

[YumengHe]

When try to run python demo.py -c configs/3_TF-Diff/text-eval.yaml
It ask for /ssd1/dengzhidong/.sym/final/ArtFormer/train_root_dir/Diff/checkpoint/05-17-09PM-12-03/diffusion-epoch=19999-loss=0.02376.ckpt

[ShuYuMo2003]

Hiii

Please follow the description from TryDemo. And donwload the checkpoint from the google drive url I provided.

Thank you.

[ShuYuMo2003]

Hiii,

Thanks for your interest in our work and for trying it out. I’m very sorry for my delayed reply, and also for not understanding what your error message meant earlier.

The cause of this error is that, although I provided the final model (TransDiffusionCombineModel) weights (which also include the diffusion weights), the constructor of TransDiffusionCombineModel is invoked before loading all model weight:

ArtFormer/model/Transformer/eval/__init__.py

Line 38 in 6153941

self.model = TransDiffusionCombineModel.load_from_checkpoint(eval_config['checkpoint_path'])

The constructor includes logic to load the diffusion weights:

ArtFormer/model/Transformer/__init__.py

Line 44 in 6153941

self.diffusion = Diffusion.load_from_checkpoint(config['diffusion_model']['pretrained_model_path'])

In fact, at this stage we only need to construct the diffusion model, and it should not require the diffusion checkpoint to be loaded there (the weights will be overrided in load_from_checkpoint).

So the fix is to replace the diffusion-weight-loading part:

ArtFormer/model/Transformer/__init__.py

Line 44 in 6153941

self.diffusion = Diffusion.load_from_checkpoint(config['diffusion_model']['pretrained_model_path'])

with code that constructs the diffusion model from the config file. You can refer to

ArtFormer/model/Diffusion/__init__.py

Line 52 in 6153941

self.sdf = SDFAutoEncoder(sdf_config_content)

, where the diffusion model loads the SDF model, for an example of how to do this within the diffusion model.

This issue did not occur on my local machine (because the diffusion model weights already existed locally), so I didn’t realize it would cause trouble for reproducibility. I’m sorry for the oversight.

Again, thank you for your interest in our work.