This research work explores the dynamics of a novel three-species predator–prey model with one prey and two predator populations within both non-diffusive and self-diffusive environments. The local stability of the model at ecologically significant equilibrium points has been assessed and the condition for the global stability of the predator-free equilibrium point has been derived. Rigorous mathematical analyses, including the assessment of solution existence, uniqueness, and boundedness, are conducted to provide a comprehensive understanding of the model’s behavior. The investigation also involves the calculation of Lyapunov exponents for the non-diffusive system, considering various fractional orders. To substantiate the theoretical findings, extensive numerical simulations are performed utilizing the highly effective Adams–Bashforth–Moulton method and the generalized method of lines. In the non-diffusive context, the system’s equilibrium maintenance reveals adaptive behaviors and resource allocation mechanisms. This sustains stability across various interactions and fractional-order settings. In the self-diffusive system, high-density spots in two-dimensional spatial distributions for all three populations indicate ecological niches and intense interactions. Three-dimensional analysis shows that fractional-order \(\alpha \) influences prey–predator dynamics. For the prey population, high-density patches shift within the region with changes in \(\alpha \) values, while predator distributions remain more consistent. These findings emphasize complex ecological interactions and stress the role of spatial dimensions and fractional order of the system in a better understanding of real-world ecosystems.