Federated Learning (FL) is a technical alternative for achieving collaboration-aware privacy-preserving Machine Learning (ML); however, modeling heterogeneous FL is a challenging issue as clients often encounter the scenario of training various local ML models due to varied data and tasks. To be specific, unexpected behaviors of malicious clients, e.g., sending erroneous updates data to the server, may threaten the training effectiveness of FL systems. In this paper, we propose a heterogeneous FL approach to defend against malicious clients from the perspective of strengthening robustness. We directly align the public data with the model feedback and re-distribute the contribution of mutual learning from the collaborative training process, in order to improve the adaptability in various contexts and eliminate the negative impacts on the accuracy from malicious clients. Our experiments have demonstrated that the proposed approach has a superior performance in both model alignment and data heterogeneity. The impact from malicious data during the training process can be effectively eliminated and the model accuracy of benign clients can be successfully maintained.