The widespread application of the Internet of Things (IoT) has brought tremendous convenience to our lives, but it has also led to a significant demand for traffic anomaly detection. Existing methods for IoT traffic anomaly detection, although achieving high detection accuracy, mostly rely on supervised learning techniques, facing challenges such as difficulty in obtaining labels and limited scalability. To address this issue, we propose a contracted autoencoder combined with Mahalanobis distance (MCAE) for unsupervised anomaly detection of IoT traffic. MCAE considers the correlations between different features and their respective importance, optimizing the measurement of reconstruction errors. Additionally, a Mahalanobis distance metric is incorporated into the loss function, resulting in a 5% to 15% improvement in accuracy compared to the original Euclidean distance. Furthermore, the model introduces an adaptive anomaly threshold selection method based on Gaussian kernel density estimation, further enhancing the sensitivity and accuracy of anomaly detection. Experimental results on the Bot-IoT and N-BaIoT datasets demonstrate that the comprehensive performance of MCAE surpasses the state-of-the-art unsupervised anomaly detection models, achieving accuracy rates of 98.55% and 99.02% respectively.