In response to the continuous emergence of novel unknown malicious traffic and the limitations of traditional detection methods, this paper presents an unknown-category malicious traffic detection approach based on contrastive learning. The proposed method utilizes contrastive learning to train an encoder for distinguishing normal traffic and multiple types of known malicious traffic, thereby enhancing inter-class separability. Subsequently, the model is fine-tuned to optimize the representations of malicious traffic categories with limited samples, amplifying the separation between different classes in the feature space. This ensures that the distance between unknown malicious traffic and other categories exceeds a predefined threshold, thereby achieving effective detection. Experimental results on the CICIoT2023 and UNSW-NB15 datasets demonstrate the superior performance of the proposed method, achieving multi-class F1-scores of 93.23 and 89.93, respectively, outperforming traditional approaches. Additionally, the method achieves an accuracy of over 83% in simulated scenarios involving unknown-category malicious traffic detection.