<p>Excavation of deep rock masses induces significant post-peak volumetric dilatancy, impacting surrounding rock mass stability and support design. In elastoplastic theory, the plastic potential function governs plastic strain direction and characterizes plastic deformation, which is crucial for describing dilatancy development. Plastic potential functions in rock mechanics are mostly defined by a dilation angle; however, models using a constant dilation angle often lack simulation accuracy, while those adopting a variable dilation angle may encounter theoretical inconsistencies and potential numerical errors. Regarding this issue, a plastic strain increment ratio index is introduced in this study, and a state-dependent plastic potential function is formulated based on its variation with equivalent plastic shear strain under different confining pressures. It is further incorporated into a Mohr–Coulomb-based strain-softening constitutive model and evaluated through triaxial numerical simulations. Comparison of the proposed model with the traditional strain-softening model and experimental data shows significantly improved accuracy in capturing post-peak dilatancy behavior. In addition, the effects of key parameters in the proposed plastic potential function on rock dilatancy under varying confining pressures, together with their impact on strain evolution paths, are systematically investigated. Finally, the applicability of the proposed plastic potential function in capturing the deformation characteristics of the surrounding rock mass under different stress conditions is validated through tunnel excavation simulations. These results provide significant theoretical support for predicting fracture deformation and assessing failure severity during deep rock engineering excavation.</p>