<p>Structural instability induced by fault slip in coalbed methane mining has garnered significant attention. Roughness plays a critical role in determining the damage mechanisms of slip surfaces. Comprehensive analysis of roughness tests, slip experiments, and damage characterization were conducted on fault-region tectonic coals. The influence of convex body height (RMS) on stress response, as well as macroscopic and microscopic damage, was systematically examined. Using the DoseResp function, predictive equations for fracture thresholds were developed. The results reveal that slip instability arises from progressive damage, transitioning from end abrasion and crack propagation to the formation of multiple fractures. Increasing RMS elevates peak shear stress and compressive deformation prior to slip, amplifies shear stress drop during stick–slip, and heightens the risk of structural fracture and instability. Notably, as RMS increases, stress drop, crack width, and damage degree align with the DoseResp function, exhibiting an initial rise followed by stabilization. At three critical roughness thresholds (RMS₀₁, RMS₀₂, and RMS₀₃), stress drop, fractures, and instability become evident on the slip surface. These findings propose a practical approach to mitigating fault instability, thereby supporting advancements in coalbed methane mining.</p>