<p>Utilizing physics-informed neural networks (PINN) for solving differential equations has gained increasing attention in real-world applications, but it is still unclear how the network parameters change during training and affect the learning accuracy and convergence rate. In this paper, we start with a toy example to find that some parameters always keep active during the total training and are named by the primary parameters, while others remain almost unchanged after a finite number of iterations and are called the secondary parameters which can be further divided into different levels. Moreover, freezing the secondary parameters stepwise and updating the primary parameters make the training in low-dimensional subspace perform better than the full parameter training. Thus, we first propose a stepwise parameter freezing strategy to enhance the performances of PINN (SPF-PINN) which highlights the positions of primary parameters by freezing the secondary parameters, and then analyze it by the neural tangent kernel theory, where the evolution of the NTK eigenvalues is employed to characterize the accelerated convergence behavior. Consequently, the SPF-PINN increases the convergence rate and improves the prediction accuracy, even solves the problems which cannot be trained by the vanilla PINN and the adaptive activation function method. Numerical results on solving different types of differential equations demonstrate the superiority of the SPF-PINN method.</p>