Under three-dimensional stress conditions, the rock strength presents a pronounced nonlinear dependence on the minor principal stress \(\sigma_{3}\) , and is significantly influenced by the intermediate principal stress (i.e., the \(\sigma_{2}\) effect). Moreover, the mechanical strength of various rock types exhibits remarkable variability. Based on the principle of energy transformation and conventional triaxial test results of 26 types of rocks, a logarithmic nonlinear strength criterion is proposed to simultaneously capture the tensile and compressive strength properties of rocks. The predictions of this strength criterion are in well agreement with the test data, and significantly superior to those of the classic strength criterion (e.g., the modified Mohr–Coulomb (MM–C) criterion, the Drucker–Prager (D–P) criterion, and the Hoek–Brown (H–B) criterion). Taking into account the enhancing and weakening effect of \(\sigma_{2}\) on the rock strength, the logarithmic strength criterion proposed above was extended to develop a three-dimensional (3D) strength criterion for rocks. When \(\sigma_{3}\) remains a constant, the proposed 3D strength criterion accurately captures the variation that rock strength shows an initial increase followed by a decrease with increasing \(\sigma_{2}\) . The predictions of this 3D strength criterion are also in good agreement with the true triaxial test results. In addition to the uniaxial tensile and compressive strengths of rock materials, the proposed 3D strength criterion incorporates three physically interpretable parameters. In the three-dimensional principal stress space, the shape of the yield surface of the novel 3D strength criterion is not fixed, but depends on the values of the three parameters, exhibiting a well flexibility and universality.