Improving Convolutional End-to-End Memory Networks with BERT for Question Answering

Question answering (QA) systems process natural language queries in order to retrieve relevant answers from data, document corpus or the Web. Memory networks have shown encouraging results in certain reasoning tasks of QA such as the end-to-end memory networks. In this paper, we explore the integration of BERT (Bidirectional Encoder Representations from Transformers) with Convolutional End-to-End Memory Networks to improve QA performance. It is anticipated that BERT will provide rich contextual embeddings, allowing for a comprehensive understanding of semantic relationships within sentences and questions. Specifically, we propose an incorporation of BERT, a state-of-the-art pre-trained language model with the Convolutional End-to-End Memory Networks multi-hop reasoning model to improve the overall QA performance. The proposed model can be fine-tuned on smaller datasets, effectively handling overfitting issue. Our experiment shows that the proposed model exhibits remarkable performance, outperforming top results achieved by other memory networks models on the Facebook ‘bAbI 1k’ dataset with an accuracy of 92.86.