Large language models (LLMs) trained on massive publicly accessible data excel in general domain tasks. However, the performance of LLMs in specialized domains such as healthcare and mathematics frequently fails to meet established benchmarks. The primary obstacle to improving this performance is the scarcity of high-quality and domain-specific data due to stringent data privacy regulations. Federated learning, which facilitates the training of LLMs across distributed devices through a decentralized approach that precludes centralized data aggregation presents a potential solution. Nevertheless, federated learning for instruction tuning encounters challenges including data heterogeneity and substantial computational resource requirements. Consequently, this paper proposes FEDNPAIT, an innovative federated learning framework for instruction tuning that incorporates Nesterov-accelerated momentum and a new parameter for controlling the adaptive strength of gradient updates during optimization to improve the performance of LLMs in heterogeneous federated learning environments. Furthermore, FEDNPAIT effectively addresses computational resource limitations through the incorporation of Low-Rank Adaptation. Experimental results indicate that FEDNPAIT effectively mitigates the adverse effects of data heterogeneity in federated environments and outperforms existing federated learning algorithms. Specifically, the results from tuned Llama3-8B demonstrate enhanced generalization and robustness, illustrating how to incorporate complex optimization algorithms into federated training of LLMs, and opening new possibilities for federated instruction tuning in sensitive industries.