how to get the impact of the words in the sentence on the emojis ?

[n0obcoder]

I am just curious that how can I get the impact of the words in the sentence that determines the respective emojis for the sentence, as shown in http://deepmoji.mit.edu/

Thanks in advance,
n0obcoder : D

[setu4993]

@n0obcoder : It should be possible by using the attention vector that the model can return. Use the return_attention=True flag when loading the model. Example: model = torchmoji_emojis(PRETRAINED_PATH, return_attention=True)

[vidyap-xgboost]

@n0obcoder : It should be possible by using the attention vector that the model can return. Use the return_attention=True flag when loading the model. Example: model = torchmoji_emojis(PRETRAINED_PATH, return_attention=True)

@setu4993 Hi, I tried this example and it throws me an error which is below:

Traceback (most recent call last):
File "examples/score_texts_emojis.py", line 60, in
ind_top = top_elements(t_prob, 5)
File "examples/score_texts_emojis.py", line 33, in top_elements
return ind[np.argsort(array[ind])][::-1]
IndexError: index 43 is out of bounds for axis 0 with size 7

What would be the solution for this? TIA.

[setu4993]

@vidyap-xgboost : Not sure why that would be caused. I could run that and get the attention vector. Do you mind sharing a more detailed code sample or Colab, and a little more about your environment?

[vidyap-xgboost]

@vidyap-xgboost : Not sure why that would be caused. I could run that and get the attention vector. Do you mind sharing a more detailed code sample or Colab, and a little more about your environment?

All I did was add the return_attention=True in /score_texts_emojis.py file.

@setu4993 This is my Google Colab notebook

[setu4993]

@vidyap-xgboost: I see the problem. Once you add return_attention=True on line 44, when the model runs through the forward pass on line 48, it is returning a tuple of (probabilities, attention_weights) instead of only probabilities that the for loop is expecting. Moreover, the for loop does not really interpret or use the attention weights, so wouldn't help much. This snippet works for me to get the attention weights and the probabilities:

test_sentence = "This is a test sentence"
model = torchmoji_emojis(PRETRAINED_PATH, return_attention=True)
probabilities, attention_weights = model(st.tokenize_sentences([test_sentence])[0])

[setu4993]

@vidyap-xgboost : If you want to continue using score_texts_emojis.py, on line 48, replace prob = model(tokenized) with prob, attention_weights = model(tokenized) after adding the return_attention=True attribute to the model setup on 44.

[vidyap-xgboost]

@vidyap-xgboost: I see the problem. Once you add return_attention=True on line 44, when the model runs through the forward pass on line 48, it is returning a tuple of (probabilities, attention_weights) instead of only probabilities that the for loop is expecting. Moreover, the for loop does not really interpret or use the attention weights, so wouldn't help much. This snippet works for me to get the attention weights and the probabilities:

test_sentence = "This is a test sentence"
model = torchmoji_emojis(PRETRAINED_PATH, return_attention=True)
probabilities, attention_weights = model(st.tokenize_sentences([test_sentence])[0])

@setu4993 I tried both approaches and the first approach gave me a tensor object when I printed. Thank you for that! Now I understand which word carries what weight in predicting the output.

However, as I am new to coding, I'm finding it a bit difficult to understand how I can write the attention_weights result to the CSV file. What should I change in the code in order to have the results written in the CSV?

Thank you once again.

[setu4993]

@vidyap-xgboost : It depends on how you want to use it after exporting to CSV, what you want to store in the CSV, etc. Try looking at the Python documentation here for CSV operations. If you want to use it for non-PyTorch cases, you might want to store the tensors as numpy arrays or lists.

[vidyap-xgboost]

@vidyap-xgboost : It depends on how you want to use it after exporting to CSV, what you want to store in the CSV, etc. Try looking at the Python documentation here for CSV operations. If you want to use it for non-PyTorch cases, you might want to store the tensors as numpy arrays or lists.

I want to write every sentence I give it to a CSV file along with emojis and their attention weights. Thanks for your help. I will take it from here. :)

[bfelbo]

Stumbled across this old issue. For future reference, the attention weights aren't that useful to getting the important words. It works better to do the sentence prediction w/o each word and see the difference, see more details here: https://huggingface.co/spaces/Pendrokar/DeepMoji/discussions/1#65eb375cdf813b9c15308c3c