Internet of Things (IoT) devices are capable of collecting data from various modalities while existing single-modal Federated Learning (FL) methods fall short of addressing the needs of cross-modal training. Cross-modal FL is a promising approach for distributed machine learning with multimodal data, enabling the joint training of models by sharing model parameters rather than raw data. However, sharing model parameters brings the risk of privacy leakage or local data revealment due to potential inference attacks. Protecting privacy of client models while maintaining high accuracy of the global model in cross-modal FL tasks remains a significant challenge. In this paper, we propose a lightweight \(\underline{P}\) rivacy- \(\underline{P}\) reserving \(\underline{C}\) ross- \(\underline{M}\) odal \(\underline{Fed}\) erated Learning (PPCM-Fed) in IoT. PPCM-Fed incorporates Differential Privacy (DP) into the client model to protect client model privacy during training. During the model aggregation process, the server evaluates the performance of each client model using a clean dataset and employs a weighted aggregation strategy to increase the contribution of high-quality models, in order to enhance the global model’s performance. By combining DP with weighted aggregation, we have achieved a better balance between privacy and performance of the global model with a low computational overhead, enhancing the practicability in IoT. Extensive experiments conducted on multi-modal datasets validate the effectiveness of our scheme.