Federated learning aims to tackle the “isolated data island” problem, where it trains a collective model from physically isolated clients while safeguarding the privacy of users’ data. However, supervised federated learning necessitates that each client labels their data for training, which can be both time-consuming and resource-intensive. Moreover, the training and transmission of deep models present challenges to the computation and communication capabilities of the clients. To address these two inherent challenges in supervised federated learning, we propose a novel lightweight unsupervised federated learning approach that leverages unlabeled data on each client to perform lightweight model training and communication by harnessing pretrained vision-language models, such as CLIP. By capitalizing on the zero-shot prediction capability and the well-trained image encoder of the pre-trained CLIP model, we have carefully crafted an efficient and resilient self-training approach. Additionally, to address data heterogeneity within each client, we propose a class-balanced text feature sampling strategy for generating synthetic instances in the feature space to support local training. The experimental results demonstrate that our proposed method greatly enhances model performance in comparison to CLIP’s zero-shot predictions and even outperforms supervised federated learning benchmark methods given limited computational and communication overhead.