With the widespread application of encryption technology in network traffic, traditional traffic classification methods face significant challenges when dealing with encrypted communications. Moreover, the effectiveness of machine learning-based classification approaches is often hindered by data isolation across different organizations and the scarcity of labeled traffic samples. To address these challenges, an encrypted traffic classification method based on multimodal asynchronous federated learning is proposed in this paper, named MaFT. This approach leverages federated learning to enable collaborative model training across distributed organizations without sharing sensitive raw data, thereby overcoming data isolation while preserving data locality requirements. The method employs multimodal learning to fully utilize different types of traffic features, including raw bytes and packet length sequences, through Deep Canonical Correlation Autoencoders that can handle both single-modal and multi-modal clients with varying data capabilities. MaFT adopts a semi-supervised training strategy, conducting unsupervised autoencoder training on the client side to reduce reliance on labeled data, while performing supervised fine-tuning on the server side to complete the traffic classification task. Additionally, an asynchronous update mechanism with staleness tolerance is introduced, combining exponential temporal decay weighting and modal importance differentiation to handle delayed model updates, thereby improving training efficiency without requiring strict synchronization. Experiments conducted on the publicly available dataset demonstrate that MaFT achieves superior performance compared to baseline methods, with accuracy reaching 98.14%, indicating the effectiveness of multimodal federated learning in addressing practical challenges of distributed traffic classification.