Not an issue: Queries regarding MATTNET CODE

[BelaChakraborty]

Hello

My name is Bela Chakraborty, an M.Phil student(pursuing) under A/PR Dr. Lei Wang at the University of Wollongong, NSW. I am very much interested in your MAttnet work and I also working on a similar referring expression problem.

I was working on your code which was released on GitHub: https://github.com/lichengunc/MAttNet

There are few things I am not able to understand, I know you would have a busy schedule, but I would really appreciate your guidance, I am a beginner and I really am very interested in your work. Would you be kind enough to guide me through the following issues:

Your paper says: we use a bi-directional LSTM to encode the context for each word. We first embed each word ut into a vector et using a one-hot word embedding, then a bidirectional LSTM-RNN is applied to encode the whole expression.

Could you kindly advise whether the "labels" used in this is the data from data_h5? Or what kind of preprocessing is done before it being passed, I am a bit lost in the code, could u direct me the exact code function?

In your code, I am a bit unclear about the "labels" being passed to the language encoder: https://github.com/lichengunc/MAttNet/blob/master/lib/layers/joint_match.py
On line 177: context, hidden, embedded = self.rnn_encoder(labels) where labels: (n, seq_len)

I just wanted to know specifically about the attribute prediction branch. I see the subject encoder code, I do understand it but it does use the weighted phrase context from RNN encoder but I am not sure what exactly is being passed into the Text model. and I was wondering how it's just trained for attributes and how the corresponding targets are made?
Thank you

Regards
Bela

If anyone has a good understanding of the flow of code, I would be grateful if they could also help me a bit in this.