<p>The exploitation of hydrocarbon resources is gradually advancing into ultra-deep formations. Currently, the understanding of the strength characteristics of rocks under ultra-high temperature (exceeding 150&#xa0;°C) and ultra-high in situ stress (exceeding 100&#xa0;MPa) conditions in ultra-deep formations remains insufficient, especially regarding the brittle–ductile transition behavior and the effects of confining pressure and temperature on peak strength. A strength analysis of deep carbonate rocks from the Tarim Basin in Xinjiang, China, revealed that the linear Mohr–Coulomb strength criterion is inapplicable under ultra-deep conditions. To accurately describe the strength characteristics of rocks in ultra-deep environments, a nonlinear Power-Function strength criterion was adopted and implemented in FLAC^3D. Then the Newton–Raphson iterative algorithm was used to improve the original algorithm in FLAC^3D, enabling the fast and high-precision solution of nonlinear constitutive equations. Triaxial compression numerical simulations validated the applicability and accuracy of the nonlinear Power-Function strength criterion and the updated algorithm within FLAC^3D. Finally, the strength criterion was further applied to a borehole stability analysis of a deep exploratory well. The results demonstrated its feasibility and potential for engineering applications. This research provides a theoretical and numerical reference for addressing the challenges of rock strength instability in ultra-deep formations.</p>