<p>The Transformer architecture has become widely adopted due to its demonstrated success, attributed to the attention mechanism at its core. Despite its successes, the attention mechanism in Transformers is associated with two issues: rank-collapse and gradient vanishing, which may hinder the Transformer’s potential in downstream performance. In this paper, we present a theoretical analysis that it is inherently difficult to address both issues simultaneously in the conventional attention mechanism. To address these issues simultaneously, motivated by generalized probability concept, we propose a generalized probabilistic attention mechanism (GPAM) and its dual-attention implementation within the Transformer architecture. Based on theoretical advantages toward both issues, GPAM extends the conventional attention mechanism’s conditions on the normalized attention scores to allow both positive and negative values while preserving a total sum. Furthermore, we empirically validate the theoretical advantages, demonstrating the superiority of daGPAM compared to other alternative attention mechanisms that were proposed to address the same issues. Additionally, we demonstrate the practical benefits of daGPAM in natural language processing tasks, such as language modeling and neural machine translation.</p>