While federated learning is a state-of-the-art framework for distributed privacy-preserving machine learning, it faces significant challenges in scenarios with non-homogeneous data. Data heterogeneity causes training instability and slows convergence. Existing reparameterization and knowledge distillation solutions introduce new challenges, such as controlling client drift and avoiding knowledge interference. In this paper, we propose a federated learning method based on QR factorization (FedQR) to address these issues. FedQR reparameterizes local model updates by projecting gradients onto the orthogonal basis of the global weights. This approach effectively mitigates client drift while seamlessly retaining the knowledge already acquired by the global model. Our extensive experiments on common federated learning vision and text datasets demonstrate that FedQR can improve communication cost by \(7.6\times \) and increase accuracy by up to \(10\%\) compared to state-of-the-art methods, particularly in non-i.i.d. scenarios. Extended evaluations on real-world feature shift datasets further confirm the robustness and performance of our method, with accelerated convergence of up to \(20 \times \) .