Some Improvement to Implement #4
Description
There are several tricks to implement to improve the performance and they won't be added util Sep. 19 for some more important projects. I opened this issue to avoid forgetting it. :)
Here are some tricks I think could be useful.
- Change the way to calculate yes/no probability. Current implementation is so naive and summarizing the final passage representation into vector and using bilinear product with question vector will be more robust. In other words, this could be seen as a simple NLI problem. This could also be used to calculate the probability for unanswerable questions in SQuAD2.0.
- Add layers of LSTM.
- There is a more efficient way to use ELMo, which could be find in Allennlp's issue. The elmo class could calculate the CNN representation for all words in the vocabulary of train set before training, because the representation for a single word is the same all the time and it don't need to be calculated every time before being passed into LSTM. This could reduce the memory usage and enhance the training speed.
- Plausible answer predicting loss(Multi-task training) could be added. For more detail, please see Read + Verify: Machine Reading Comprehension with Unanswerable Questions
- More manual features, for example, lemma of words, exact match feature, pre-align embedding.