This paper presents a groundbreaking solution to the critical challenge of outlier sensitivity in transformer-based architectures for multi-round dialogue systems. Our research introduces the Outlier-Aware Transformer (OAT), a sophisticated architecture that revolutionizes dialogue processing through innovative residual attention compensation mechanisms and context-aware fine-tuning protocols. The framework demonstrates exceptional capabilities in managing extended conversational sequences, significantly reducing error propagation across multiple dialogue turns. Through extensive experimentation on diverse dialogue datasets, we establish that OAT substantially outperforms conventional transformer architectures in maintaining contextual coherence, reducing outlier-induced errors by 37%, and improving response relevance by 42%. This work represents a significant advancement in dialogue system architecture, addressing fundamental limitations in existing approaches while establishing new benchmarks for performance and reliability in multi-round conversation management.