In the field of flexible manufacturing, the multi-objective flexible job shop scheduling problem, which considers multiple economic performance optimization indicators such as energy consumption, has been receiving increasing attention. Evolutionary algorithms have been proven to be efficient and widely used methods for solving multi-objective flexible job shop scheduling problems. However, in most existing research, the key parameters of the evolutionary operations in these algorithms are determined using empirical methods and adaptive adjustments, which ignore the population information from previous iterations. This can lead to a degradation in the evolutionary performance when the evolution is unstable. To address this issue, this study proposes an improved NSGA-III (RLNSGA-III) to simultaneously optimize the makespan, energy consumption, total machine load, and tardiness indicators in many-objective flexible job shop scheduling problems. The main improvements are as follows: (1) introducing data mining methods to improve the quality of population initialization, (2) constructing a neighborhood search structure based on critical paths to enhance the local search capability of the algorithm, and 3) designing a reinforcement learning-based method to automatically learn and adjust the crossover and mutation probability parameters, thereby achieving a better balance between global and local search capabilities and improving the convergence ability of the algorithm. The experimental results on multiple test cases compared with classical algorithms demonstrate that the proposed algorithm has superior performance in solving multi-objective flexible job shop scheduling problems.