Federated Learning (FL) faces substantial challenges due to heterogeneous label noise, which can hinder the effectiveness of collaborative training. Most existing methods assume that label noise is independent of features, while real-world scenarios frequently involve instance-specific annotation errors influenced by task complexity. These issues are compounded by noise heterogeneity and class imbalance. In this paper, we address the more challenging issue of instance-dependent label noise in FL and propose a novel framework, FedIDN. This framework utilizes class-wise confidence values to improve noise identification, even in scenarios with class imbalance. We initially detect noisy clients based on class-wise confidence variance. Then, we employ a global confidence threshold combined with loss metrics to distinguish clean samples from noisy ones within noisy clients, subsequently applying confidence-regularized loss and negative cross-entropy loss to update these clients. Finally, the global model is refined using a noise-aware aggregation function to ensure robust and effective learning. Experimental results across various noise types and ratios demonstrate that our method consistently outperforms existing techniques, underscoring its effectiveness in confronting instance-dependent label noise in FL.