The Yellow River Guxian Water Conservancy Hub Project is located in the southern part of the Jin-Shaanxi Gorge section of the northern main stream of the Yellow River, which is a relatively uplifted mountainous plateau area, with Lüliang Mountain on the east side and Weibei Mountain on the west side. Considering the failure mode of rock mass during the analysis of anti-slip stability of the dam foundation, indoor test and in-situ test were used to carry out comparative indoor test and in-situ test for different lithological sandstones in the flat of the bottom of the dam foundation. The physical and mechanical indexes and shear strength parameters of different lithological rock masses were analyzed and it was found that the stronger the water absorption in different rock masses, the smaller the elastic modulus and saturated uniaxial compressive strength of the rock mass, and the moisture content was inversely proportional to the softening coefficient. The shear strength parameters (friction coefficient and cohesion) of the rock block direct shear test were 1.05–1.24 times and 3.72–4.58 times of the shear strength parameters of the rock mass direct shear test. According to different preset shear angles, the failure of direct shear strength of rock mass is further analyzed, which belongs to the brittle ~ plastic composite failure mode, which provides reliable design parameters for the analysis of anti-slip stability of dam body.