<p>Rock shear failure-induced instability, characterized by pronounced nonlinearity and abrupt transitions, frequently leads to severe geological hazards in deep resource extraction and rock engineering structures. This study uses the critical phase transition theory and catastrophe theory to investigate the acoustic emission (AE) characteristics associated with the transition from stable crack propagation to dynamic shear instability. Real-time AE monitoring was performed during direct shear tests on sandstone to analyze AE responses during the evolution from microcracking to through-going fracture. The results show that during the instability stage, AE energy release, event count, and amplitude increased markedly, whereas fluctuations in AE parameter values decreased, indicating enhanced crack interactions and a sudden shift in failure mode during nonlinear instability. The <i>b</i>-value derived from the maximum likelihood method exhibited a significant decline, reflecting the rapid development of large fractures and the onset of instability. The variance and autocorrelation coefficient of AE energy and count exhibited a sharp increase immediately before instability. As the normal stress increased, the multifractal spectrum width (Δ<i>α</i>) of AE energy and count gradually decreased, suggesting that high-energy AE events increasingly dominated destabilization. A novel early warning model based on swallowtail catastrophe theory was developed to overcome the limitations of conventional instability warning methods. This model accurately captures the nonlinear evolution of AE parameters and provides high predictive accuracy and engineering applicability. It is superior to existing models that use the <i>b</i>-value, variance, and autocorrelation coefficient as damage precursors because they exhibit only slight fluctuations before instability.</p>