<p>The relative density is a key indicator of the mechanical properties of coarse-grained soil. To better explain the shear behaviors such as dilatancy–contractancy and softening–hardening in simulations based on Discrete Element Method (DEM), the definitions of the maximum void ratio, minimum void ratio, and relative density were generalized to consider the impact of the confining pressure during the shear process. Comprehensive tests were conducted based on direct shear tests through the DEM. Preparation methods were presented to obtain specimens with the maximum void ratio, minimum void ratio, and target relative density. The direct shear tests show that the shear behavior of coarse-grained soil significantly varies with changes in relative density. Specifically, under high relative density conditions, soils tend to exhibit dilatancy behavior accompanied by softening characteristics. Conversely, under low relative density conditions, soils are more prone to contractancy and display hardening behavior. The peak shear strength increases with increasing relative density, indicating enhanced resistance to deformation. Additionally, compared with the void ratio, the relative density provides a more accurate description of the variation of shear strength parameters with the degree of compactness. These findings suggest that the interactions between soil particles, influenced by confining pressure and relative density, play a crucial role in determining the macroscopic shear behavior of these materials. This study provides a theoretical basis for density control and strength evaluation of coarse-grained soil.</p>