A Hybrid and Inexact Algorithm for Nonconvex and Nonsmooth Optimization

The problem of nonconvex and nonsmooth optimization (NNO) has been extensively studied in the machine learning community, leading to the development of numerous fast and convergent numerical algorithms. Existing algorithms typically employ unified iteration schemes and require explicit solutions to subproblems for ensuring convergence. However, these inflexible iteration schemes overlook task-specific details and may encounter difficulties in providing explicit solutions to subproblems. In contrast, there is evidence suggesting that practical applications can benefit from approximately solving subproblems; however, many existing works fail to establish the theoretical validity of such approximations. In this paper, the authors propose a hybrid inexact proximal alternating method (hiPAM), which addresses a general NNO problem with coupled terms while overcoming all aforementioned challenges. The proposed hiPAM algorithm offers a flexible yet highly efficient approach by seamlessly integrating any efficient methods for approximate subproblem solving that cater to specificities. Additionally, the authors have devised a simple yet implementable stopping criterion that generates a Cauchy sequence and ultimately converges to a critical point of the original NNO problem. The proposed numerical experiments using both simulated and real data have demonstrated that hiPAM represents an exceedingly efficient and robust approach to NNO problems.