Improving Natural Language Inference with Residual Attention

Natural language inference (NLI) aims to identify the logical relationship between a premise and a corresponding hypothesis, which requires the model should have the ability of effectively capturing their semantic relationship. Most of the existing transformer-based models tend to concatenate the premise and hypothesis together as the input of the model and capture their relationship through multi-head self-attention mechanism, which as a result might only consider their plain context-sensitive relationship and neglect the potentially mutual impacts of their contextual semantics. To better model the relationship between the premise and hypothesis, we propose a new transformer-based model RAN4NLI that consists of a sequence encoder based on pre-trained language model for encoding the input semantics and an interaction network based on residual attention for further capturing their relationship. We utilize residual attention for combining multi-head self-attention and cross-attention information so as to strengthen the potential semantic relationship between the premise and hypothesis. Experiments conducted on two canonical datasets, SNLI and SciTail, demonstrate that our RAN4NLI achieves comparable performance with other strong baseline models.