Result of eval_explanation_label.sh

[chandru4ni]

Hi Lisa,

We ran the script eval_explanation_label.sh and got the json file created in below path.
generated_sentences/birds_fg_test/gve_models/explanation-label_1006_pred/beam1/generation_result.json

We wanted to know if this is final output for explanation labels for the birds data set you have considered. And we need to extract the labels from this json to display them in a more readable format. Or is there any additional step which we are missing to generate explanation labels? Please advice.

Rgds,
Chandra.

[chandru4ni]

We have seen the json output and it contains the labels for each image in the data set.

Now we want to train the model with our data set and as per our inspection we need three files for training the model.

1. data/vocab.txt
2. data/descriptions_bird.train_noCub.fg.json
3. data/description_sentence_features.p

As of now you have stored these files in your path and using them and allowing us to download. But since want to train the model for our dataset we want to know how to generate the description_sentence_features.p file. Please suggest.

Is there anything we need to have before we train your model with our data set? We will continue to look for details from our end.

[LisaAnne]

Hi! description_sentence_features.p should just be a label embedding for different classes. You can just use one hot vectors (performance is about the same and it will be easier to create). Cheers, Lisa

…

On Thu, Feb 22, 2018 at 4:11 AM, chandru4ni ***@***.***> wrote: We have seen the json output and it contains the labels for each image in the data set. Now we want to train the model with our data set and as per our inspection we need three files for training the model. 1. data/vocab.txt 2. data/descriptions_bird.train_noCub.fg.json 3. data/description_sentence_features.p As of now you have stored these files in your path and using them and allowing us to download. But since want to train the model for our dataset we want to know how to generate the description_sentence_features.p file. Please suggest. Is there anything we need to have before we train your model with our data set? We will continue to look for details from our end. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#5 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFQm2iEjsqX1UCQ2vbAbIX312QXNqf_Rks5tXT05gaJpZM4SNiDj> .

[chandru4ni]

Thanks Lisa.

Yes you are right that description_sentence_features.p is a label embedding for different classes (200 X 1000 numpy array). You have told to use one of the hot vectors for this. If I am not wrong, I understand from your paper that you are using Compact Bilinear Pooling to get this vector. Though I spent some time on inspecting the code base I was not able to find the code to generate this vector even for your dataset.

So I need your help here to understand the flow since I am planning to use my own data set.

regards,
Chandra.

[chandru4ni]

Is my understanding that description_sentence_features.p is representation of features (1000) for 200 classes of birds correct?

[LisaAnne]

I think this is the code you want: https://github.com/LisaAnne/ECCV2016/blob/ECCV2016/examples/ECCV2016/make_class_embedding.sh However, it is probably easier to encode classes as one hot embeddings.

…

On Fri, Feb 23, 2018 at 11:23 AM, chandru4ni ***@***.***> wrote: Is my understanding that description_sentence_features.p is representation of features (1000) for 200 classes of birds correct? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#5 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFQm2vdHyyxQ8pQjANYU_w35wCf_w2GBks5tXvQGgaJpZM4SNiDj> .

[chandru4ni]

Thanks Lisa.
I saw the code of extract_val.py and yes it is generating the label embeddings and saving the file as pickle file. But while I ran the script to generate the pickle file, it is crashing. While debugging I observed that it is crashing while processing through the net at net.forward() line in extract_val.py.

Below are my few observations regarding make_class_embedding.sh script.
Below are the inputs to extract_val.py.
image_net=examples/ECCV2016/prototxt/deploy.prototxt
#word_net=examples/ECCV2016/prototxt/wtd_1000_all.prototxt
word_net=examples/ECCV2016/prototxt/wtd_1000.prototxt

Chandra

#model=snapshots/description_1006
model=gve_models/description_1006

I see that word_net file name is wtd_1000_all.prototxt and it does not exist in prototxt directory as part of downloaded files. But a file named wtd_1000.prototxt file is in prototxt directory, so I have changed to that file name. And the model filename is snapshots/description_1006 but I do not see any file in snapshots directory. And description_iter_1000.cafemodel is generated when we run train_description.sh in snapshots directory. We have description_1006.caffemodel downloaded from your site at gve_models. So I have changed the model to gve_models/description_1006.caffemodel

So I wanted to check if these changes are causing the crash. If yes, are we then missing wet_1000_all.prototxt and snapshots/description_1006.caffemodel files?

If not, I will continue debugging the code to see the cause of the crash. I request your thoughts on above and if possible you can throw light on the cause of the crash.

regards,
Chandra.

[chandru4ni]

So my question if the changes i have done are causing the crash? And I am unable to find the file names which you have given in the make_class_embedding.sh. Are they missing in your code base by any chance? If possible you can let me know reason of the crash for any other reason. I will continue debugging the code for further analysis.

regards,
Chandra.

[chandru4ni]

Since I have changed the file names in make_class_embedding.sh as below, i want to confirm before I proceed with further debugging. The lines in bold are the original lines and below them are modified by me. As the original files are not available to compare I want your inputs in this regard.

#word_net=examples/ECCV2016/prototxt/wtd_1000_all.prototxt
word_net=examples/ECCV2016/prototxt/wtd_1000.prototxt

#model=snapshots/description_1006
model=gve_models/description_1006

regards,
Chandra.

[chandru4ni]

I changed the script as below and used wtd_1000_20words.prototxt file rather than wtd_1000_all.prototxt file which you had used in the script. With this make_class_embedding.sh ran successfully.
word_net=examples/ECCV2016/prototxt/wtd_1000_20words.prototxt

I generated the model with generated label embeddings using train_explanation_label.sh. But when I tried to use this generated model for for evaluation script eval_explanation-label.sh, I got all captions as same for all birds and that it is not some symbols as given below.

"196.House_Wren/House_Wren_0057_187157.jpg", "caption": "" & "."}, {"image_id": "099.Ovenbird/Ovenbird_0070_93140.jpg", "caption": "" & "."}, {"image_id": "135.Bank_Swallow/Bank_Swallow_0018_129891.jpg", "caption": "" & "."}, {"image_id": "117.Clay_colored_Sparrow/Clay_Colored_Sparrow_0068_110706.jpg", "caption": "" & "."}

Looks like something wrong in the prototxt file I used or the steps I am following. Please advice with your suggestion.

regards,
Chandra.

[chandru4ni]

Corrected the comments which had some typos.

I changed the script as below and used wtd_1000_20words.prototxt file rather than wtd_1000_all.prototxt file in make_class_embedding.sh script. With this make_class_embedding.sh ran successfully.
word_net=examples/ECCV2016/prototxt/wtd_1000_20words.prototxt

I generated the model with label embeddings using train_explanation_label.sh. But when I tried to use this generated model for for evaluation script eval_explanation-label.sh, I got all captions as same for all bird images and that it has got some symbols rather than words as given below.

"196.House_Wren/House_Wren_0057_187157.jpg", "caption": "" & "."}, {"image_id": "099.Ovenbird/Ovenbird_0070_93140.jpg", "caption": "" & "."}, {"image_id": "135.Bank_Swallow/Bank_Swallow_0018_129891.jpg", "caption": "" & "."}, {"image_id": "117.Clay_colored_Sparrow/Clay_Colored_Sparrow_0068_110706.jpg", "caption": "" & "."}

Looks like something wrong in the prototxt file I used or the steps which I am following. Please advice with your suggestion.

regards,
Chandra.

[chandru4ni]

Hi Lisa,

Now I am getting the captions correctly by using the make_class_embedding.sh script and using the deploy.prototxt and wtd_1000.prototxt files. And this script requires snapshots/description_1006 model to be available. But if we want to run this script for the first time then we will not have the model available, hence as you indicated earlier I wanted to generate the label embeddings using word2vec or Grove method. The output of your make_class_embedding.sh script is 200X1000 numpy array and I assume 200 corresponds to classes of birds. In that case word2vec and Grove provides word embeddings. Hence I am thinking on how to get the class embeddings required for our explanation model. Any suggestions from your end will be helpful.

regards,
Chandra.

[chandru4ni]

Hi Lisa,

Now I am getting the captions correctly by using the make_class_embedding.sh script and using the deploy.prototxt and wtd_1000.prototxt files. And this script requires snapshots/description_1006 model to be available. But if we want to run this script for the first time then we will not have the model available, so I thought of generating the label embeddings using word2vec or Grove method. The output of your make_class_embedding.sh script is 200X1000 numpy array and I assume 200 corresponds to classes of birds. In that case using word2vec and Grove we get word embeddings and not mapped to classes. Hence I am thinking on how to get the class embeddings required for our explanation model, should I look for other tools? I will continue exploring from my end, just wrote to you so that any suggestions from your end would be helpful.

regards,
Chandra.

[LisaAnne]

Hello, You should probably retrain a model with your data and then follow the same steps I did to make my class embeddings.

…

On Wed, Feb 28, 2018 at 7:52 PM, chandru4ni ***@***.***> wrote: Hi Lisa, Now I am getting the captions correctly by using the make_class_embedding.sh script and using the deploy.prototxt and wtd_1000.prototxt files. And this script requires snapshots/description_1006 model to be available. But if we want to run this script for the first time then we will not have the model available, so I thought of generating the label embeddings using word2vec or Grove method. The output of your make_class_embedding.sh script is 200X1000 numpy array and I assume 200 corresponds to classes of birds. In that case using word2vec and Grove we get word embeddings and not mapped to classes. Hence I am thinking on how to get the class embeddings required for our explanation model, should I look for other tools? I will continue exploring from my end, just wrote to you so that any suggestions from your end would be helpful. regards, Chandra. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#5 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFQm2mCzekRrpTkXEIWNd4wYAaTQFdVrks5tZ3B7gaJpZM4SNiDj> .

[chandru4ni]

Thanks Lisa, yeah I am setting the data set to train the model. In the mean time I found that right now the code is able to provide visual explanation for all birds that belong to bilinear_pred.p master data set. If I give a new bird which does not belong to master data set then it is not identifying it correcty. Is it a prerequisite for the project to have the bird belong to only to this data set, since your research is more specific to provide visual explanations rather the classifying the image or identifying the image. But when I go for my data set, I need to first preprocess the image to get the descriptors to identify the image. Right now the current code is not working for preprocessing the image. Let me know if my understanding is correct so that I will continue my exploration on getting my image preprocessed to get descriptors through the image net.

Regards,
Chandra.

[chandru4ni]

Thanks Lisa, yeah I am working on the data set to train the model and generate the embeddings as you indicated above. In the mean time I found that right now the code is able to provide visual explanation for all birds that belong to bilinear_pred.p master data set. If I give a new bird which does not belong to master data set then it is not identifying it correctly. Is it a prerequisite for the project to have the bird belong to only to this data set, since your research is more specific to provide visual explanations rather the classifying the image or identifying the image. But when I go for my data set, I need to first preprocess the image to get the descriptors to identify the image. Right now the current code is not working for preprocessing the image. Let me know if my understanding is correct so that I will continue my exploration on getting my image preprocessed to get descriptors through the image net.

Regards,
Chandra.