Not able to reproduce the result with pretrained model provided

[ruotianluo]

I can train the model using current code and get similar result.
But when I try to load the weights using load_checkpoint option, I can bad results on guesser:

Training loss: 2.4473261008633354
Training error: 0.8252262064423064
Validation loss: 2.424793016379049
Validation error: 0.8200415782867723

I'm using the Tensorflow (1.0<->1.2) one and my tensorflow version is 1.1.0

[harm-devries]

Hi!

I'll need a bit more info to see if I can help you:

• Are you loading your own trained model or the one provided by us?
• Which python version are you using?

[krsandeep98]

Hi @harmdevries89 and @fstrub95,
Thanks a lot for the wonderful codebase. I just had a few queries about reproducing the numbers mentioned in the readme of this repository. It says the results reported here are from the checkpoint for Tensorflow 1.0 - 1.2, correct me if I am wrong.
But when I downloaded the same from http://florian-strub.com/github/pretrained_models.tf1-2.zip and used that to load the weights, I am getting an error that a particular variable is not present in the checkpoint. Find more details below (I am using Python3.7 and Tensorflow-gpu 1.15)

This is if I use the qgen_loader_supervized in line 166 in train_qgen_reinforce.py

2020-07-02 17:01:26.662091: W tensorflow/core/framework/op_kernel.cc:1651] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key qgen/image_embedding/fully_connected/W not found in checkpoint

tensorflow.python.framework.errors_impl.NotFoundError: 2 root error(s) found.
  (0) Not found: Key qgen/image_embedding/fully_connected/W not found in checkpoint
	 [[{{node save_5/RestoreV2}}]]
	 [[save_5/RestoreV2/_9]]
  (1) Not found: Key qgen/image_embedding/fully_connected/W not found in checkpoint
	 [[{{node save_5/RestoreV2}}]]

File "src/guesswhat/train/train_qgen_reinforce.py", line 166, in <module>
    qgen_loader_supervized = tf.train.Saver(var_list=qgen_var_supervized)

Traceback (most recent call last):
  File "src/guesswhat/train/train_qgen_reinforce.py", line 167, in <module>
    qgen_loader_supervized.restore(sess, os.path.join(args.networks_dir, 'qgen', args.qgen_identifier, 'params.ckpt'))
  File "/playpen/sandeep/anaconda3_installation/envs/guesswhat/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 1306, in restore
    err, "a Variable name or other graph key that is missing")
tensorflow.python.framework.errors_impl.NotFoundError: Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

If I use the continue_exp flag and try loading the weights with qgen_saver, I get a similar error

2020-07-02 16:57:52.819278: W tensorflow/core/framework/op_kernel.cc:1651] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key qgen/image_embedding/fully_connected/W not found in checkpoint

tensorflow.python.framework.errors_impl.NotFoundError: 2 root error(s) found.
  (0) Not found: Key qgen/image_embedding/fully_connected/W not found in checkpoint
	 [[{{node save/RestoreV2}}]]
	 [[save/RestoreV2/_19]]
  (1) Not found: Key qgen/image_embedding/fully_connected/W not found in checkpoint
	 [[{{node save/RestoreV2}}]]

I don't get these errors when I am using the pretrained models for TF 1.3 (http://florian-strub.com/github/pretrained_models.tf1-3.zip), but then I get much worse performance.
Here are some initial numbers when I run train_qgen_reinforce.py with those models.

Oracle test error: 0.4949441328262667
Guesser test loss: 2.847108743752645
Guesser test error: 0.8328140309661821
QGen test loss: 20.794538643709434
 New Objects  <<<
Accuracy (train - greedy): 0.15172885412659742
 New Games  <<<
Accuracy (test - greedy): 0.140718940508724

Any help would be much appreciated.

Thanks
Sandeep

[ttaoREtw]

I can train the model using current code and get similar result.
But when I try to load the weights using load_checkpoint option, I can bad results on guesser:

Training loss: 2.4473261008633354
Training error: 0.8252262064423064
Validation loss: 2.424793016379049
Validation error: 0.8200415782867723

I'm using the Tensorflow (1.0<->1.2) one and my tensorflow version is 1.1.0

I think you have to copy the dict.json (included in the pre-trained model link) they provided to data/dict.json.
The dict.json you created may not be the same as they used for training.

[fstrub95]

The vocabulary codebase is dependant of the hashing function of the python dict. However, this hashing changed between python2 and python3. Thus, if you do not use the same python version as we did when pretraining the models(I do not remember the one we used!), then it is expected to not be able to compute the results as the word idx are randomly shuffled. Thus, it is quite important to use the provided dict, and not to generate a new one.

…

On Wed, Jul 29, 2020, 4:48 PM Tao ***@***.***> wrote: I can train the model using current code and get similar result. But when I try to load the weights using load_checkpoint option, I can bad results on guesser: Training loss: 2.4473261008633354 Training error: 0.8252262064423064 Validation loss: 2.424793016379049 Validation error: 0.8200415782867723 I'm using the Tensorflow (1.0<->1.2) one and my tensorflow version is 1.1.0 I think you have to copy the dict.json they provided to data/dict.json. The dict.json you created may not be the same as they used for training. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#23 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADF7HBHTZSCRQ7MG4PXTFKLR6AZFJANCNFSM4ETT6ZRQ> .