Federated learning (FL) enables geographically distributed devices to collaboratively train models while safeguarding user privacy and data security. However, when client data heterogeneity or system heterogeneity leads to varying client training speeds, the training accuracy and efficiency of FL are greatly affected by slow clients (stragglers) processing strategies. Existing works mainly focus on mitigating the impact of stragglers by introducing straggler suppression strategies in asynchronous training. Although this approach improves training efficiency, it usually results in unbalanced data distribution and affects model accuracy. We observe that the impact of stragglers on training varies across stages: they may improve accuracy during stable stage of the model, but affect accuracy when the model is unstable. However, existing works often overlook this fact. To address this, this paper proposes an FL method with a dynamic straggler processing strategy. Specifically, it dynamically adjusts the aggregation frequency of stragglers through a clustering-based two-layer semi-asynchronous training framework (C2SA) and adjusts the aggregation strategy through a model stability-based dynamic aggregation strategy (MSDAS). Experimental results show that, compared to state-of-the-art methods, our method improves model accuracy by 3.48% to 11.45% while maintaining training efficiency.