To address the impact of parameter filter length and the number of modes on the performance of Feature Mode Decomposition (FMD), a parameter-optimized FMD fault diagnosis method is proposed. This method begins with the basic parameter settings for FMD and defines limits for the filter length L and mode number n. Subsequently, the method employs the Whale Optimization Algorithm (WOA) for iterative optimization of FMD, introducing a new health indicator as the fitness function during the WOA iterations. Finally, the fault in the rolling bearing is diagnosed using the envelope spectrum of the Intrinsic Mode Functions (IMF) obtained from decomposition. Simulation and experimental results demonstrate that the proposed new index yields significantly better diagnostic outcomes compared to other fitness functions. Additionally, when compared to Variational Mode Decomposition (VMD), the parameter-optimized FMD extracts fault feature frequencies more effectively, showcasing superior feature extraction capabilities and facilitating accurate diagnosis of rolling bearing faults.