With the rapid growth of applications such as smart cities and the Internet of Things (IoT), the exponential increase in mobile data traffic has pushed the limits of traditional communication spectrum usage, thus prompting a shift towards semantic communication. Semantic communication, which emphasizes the transmission of meaning to minimize redundant information, is central to this transition. However, existing solutions fail to address key challenges in IoT environments, such as long training and update delays for terminal models, limited computational power, restricted edge storage, and inconsistencies in semantic models across heterogeneous networks. These issues lead to poor generalization across diverse channels and ambiguities when transmitting multi-user background knowledge. To tackle these challenges, this paper proposes a cloud-edge-terminal collaborative distributed semantic communication architecture. Based on the federated learning (FL) framework, the architecture offloads the training and update tasks of the semantic communication system model to edge servers and applies FL to solve the problem of semantic collaboration in the IoT. While reducing the computational overhead of terminal devices, it enhances the adaptability and communication feasibility of the system. In addition, transfer learning (TL) is utilized to freeze the parameters of cross-domain shared modules, while fine-tuning the adaptation modules for new scenarios. This approach facilitates the rapid generalization of the model across different channels and background knowledge. Experimental results demonstrate the effectiveness and practicality of the proposed method, validating its potential for real-world applications in the field of IoT scenarios.