The low-rank decomposition model is frequently employed in defect detection. It separates the target matrix into a low-rank component and a sparse component using the nuclear norm and the \(l_1\) -norm, which aids in extracting the background and defects. However, the nuclear norm, derived from singular value decomposition, often fails to effectively extract the background of fabrics. This paper introduces a novel matrix norm, defined by integrating several key elementary functions, enhancing the separation of the low-rank and sparse matrices. The Alternating Direction Method of Multipliers (ADMM) typically solves the low-rank decomposition model with a fixed step size penalty factor. This study dynamically adjusts the penalty factor based on defect detection characteristics, thus enhancing the algorithm’s computational efficiency. Additionally, the convergence of the proposed algorithm is validated. Experimental results demonstrate that this new model not only precisely distinguishes the sparse matrix but also achieves higher computational efficiency, surpassing other existing methods in both accuracy and efficiency.