Fact verification, as a critical technology for curbing the spread of misinformation, has garnered significant attention in recent years. However, existing methods in question-answering dialogues face two major challenges. Firstly, mainstream methods rely on sequential models or static attention and are often misled by redundant information. Secondly, inadequate utilization of label semantics results in difficulties in distinguishing fine-grained categories. To address these challenges, this paper integrates nonlinear Graph Attention and Contrastive Learning for fact verification in question-answering Dialogues (DialGACL). The framework incorporates the K-GAT module, which leverages KAN-driven nonlinear attention to dynamically adjust edge weights while filtering noisy nodes, thereby constructing a deep semantic network. Additionally, to address deficiencies in label semantics, we propose a prototype contrastive loss that utilizes learnable label prototypes to enhance the discriminability of the feature space. Experimental results demonstrate that our DialGACL outperforms state-of-the-art methods on three benchmark datasets.