Scholarlink

<p>Transformer models excel in machine reading comprehension but struggle with complex reasoning tasks due to the lack of explicit relational knowledge in input sequences. Existing solutions often rely on external knowledge graphs, introducing challenges in knowledge selection, availability, and processing overhead. To address this, we propose HEAT (Heterogeneous Entity-Aware Transformer), a novel method that integrates reasoning knowledge from a document-derived heterogeneous graph directly into the transformer’s self-attention mechanism, eliminating external dependencies. HEAT employs a fusion attention pattern with three components: global-local attention for word tokens, graph-based attention prioritizing entity tokens connected in the graph, and relationship-aware attention between entity and word tokens. These are enhanced by relative position labels integrated into LUKE’s entity-aware self-attention. Evaluated on ReCoRD (commonsense reasoning), WikiHop (multi-hop reasoning), and CliCR (clinical reasoning), HEAT achieves superior performance, surpassing LUKE by 1.1%, 5.6%, and 2.2%, respectively, and outperforming LUKE-Graph, demonstrating its effectiveness in reasoning-intensive tasks.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

HEAT: a heterogeneous entity-aware transformer for efficient and accurate cloze-style machine reading comprehension

  • 期刊论文

  • 发表日期：

    2026年2月3日

查看全文
  • Shima Foolad,
  • Kourosh Kiani

摘要

Transformer models excel in machine reading comprehension but struggle with complex reasoning tasks due to the lack of explicit relational knowledge in input sequences. Existing solutions often rely on external knowledge graphs, introducing challenges in knowledge selection, availability, and processing overhead. To address this, we propose HEAT (Heterogeneous Entity-Aware Transformer), a novel method that integrates reasoning knowledge from a document-derived heterogeneous graph directly into the transformer’s self-attention mechanism, eliminating external dependencies. HEAT employs a fusion attention pattern with three components: global-local attention for word tokens, graph-based attention prioritizing entity tokens connected in the graph, and relationship-aware attention between entity and word tokens. These are enhanced by relative position labels integrated into LUKE’s entity-aware self-attention. Evaluated on ReCoRD (commonsense reasoning), WikiHop (multi-hop reasoning), and CliCR (clinical reasoning), HEAT achieves superior performance, surpassing LUKE by 1.1%, 5.6%, and 2.2%, respectively, and outperforming LUKE-Graph, demonstrating its effectiveness in reasoning-intensive tasks.