Federated Learning (FL) has gained significant traction due to its decentralized training approach while preserving privacy. The Extreme Learning Machine (ELM) emerged as a competitive technique for diverse classification and regression tasks, owing to its advantages such as fewer trainable parameters and expedited training times. Nonetheless, it suffers from overfitting and suboptimal generalization. Motivated by these considerations, we introduce FedELM-SGD, an iterative variant of ELM which integrates mini-batch stochastic gradient descent (SGD) within privacy-preserving federated learning settings. We also propose an extreme learning factory under federated settings (FedELF) to counteract the impact of random parameter initialization in ELM. In this study, we outline a two-stage methodology: firstly, the generation of privacy-preserving datasets through perturbation of real datasets using Differential Privacy techniques; secondly, the invocation of either FedELM-SGD or FedELF to construct a globally shared classification model. We demonstrate the efficacy of our proposed methods across various finance and medical-related datasets.