In this paper, we address the problem of Rationale Extraction (RE) from Natural Language Processing: given a context (C), a related question (Q) and its answer (A), the task is to find the best sentence-level rationale ( \(R^*\) ). This rationale is loosely defined as being the subset of sentences of the context C such that producing A would require at least \(R^*\) . We have constructed a dataset where each entry is composed of the four terms (C, Q, A, \(R^*\) ) to explore different methods in the particular case where the answer is one or multiple full sentences. The methods studied are based on TF-IDF scores, embedding similarity, classifiers and attention and have been evaluated using a sentence overlap metric akin to the Intersection over Union (IoU). Results show that the best scores were achieved by the classifier-based approach with the nuance of a better scaling with the attention-based method as the size of the context increases, which is a challenge for all other methods. We also show that generating A significantly decreases the performance of the attention-based method, but training the model to generate A can improve the results, linking the ability to generate with the accomplishment of the task.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Exploration of Rationale-Extraction Methods for Closed-Domain Question Answering with a New Sentence-Level Rationale Dataset

  • Lize Pirenne,
  • Samy Mokeddem,
  • Damien Ernst,
  • Gilles Louppe

摘要

In this paper, we address the problem of Rationale Extraction (RE) from Natural Language Processing: given a context (C), a related question (Q) and its answer (A), the task is to find the best sentence-level rationale ( \(R^*\) ). This rationale is loosely defined as being the subset of sentences of the context C such that producing A would require at least \(R^*\) . We have constructed a dataset where each entry is composed of the four terms (C, Q, A, \(R^*\) ) to explore different methods in the particular case where the answer is one or multiple full sentences. The methods studied are based on TF-IDF scores, embedding similarity, classifiers and attention and have been evaluated using a sentence overlap metric akin to the Intersection over Union (IoU). Results show that the best scores were achieved by the classifier-based approach with the nuance of a better scaling with the attention-based method as the size of the context increases, which is a challenge for all other methods. We also show that generating A significantly decreases the performance of the attention-based method, but training the model to generate A can improve the results, linking the ability to generate with the accomplishment of the task.