Encrypted traffic classification is essential to network optimization, quality of service improvement, and network security maintenance. However, this task has become increasingly challenging with the emergence of new applications and protocols. Existing deep learning-based methods either directly use the raw packet payloads for classification without considering the contextual relationships between packets, or use flow’s statistical features ignoring the inline relationships between packet payloads. Consequently, these approaches result in less accurate or less generalizable classification, which cannot be adapted to different classification scenarios. To address them, we propose a method called CUFT, which involves separating the irrelevant packets from the original traffic and extracting the continuous unidirectional flows (cuflows) for classification. Subsequently, we utilize the long-distance capturing capability of the multi-head attention mechanism to obtain the contextual relationships among the relevant packets in the cuflow and the inline relationships among the packet payloads. To verify the effectiveness and generality of CUFT across various classification tasks, we conducted experiments on three complex classification tasks using three publicly available datasets, ISCX-Tor, ISCX-VPN and USTC-TFC. Our results show that our proposed method achieves exceptional accuracy rates of 99.67%, 98.33% and 99.18% on these three complex classification tasks respectively.