A refined proximal algorithm for nonconvex multiobjective optimization in Hilbert spaces

This paper is devoted to general nonconvex problems of multiobjective optimization in Hilbert spaces. Based on limiting/Mordukhovich subgradients, we define a new notion of Pareto critical points for such problems, establish necessary optimality conditions for them, and then employ these conditions to develop a refined version of the vectorial proximal point algorithm providing its detailed convergence analysis. The obtained results largely extend those initiated by Bonnel et al. [SIAM J Optim, 15 (2005), pp. 953–970] for convex vector optimization problems, specifically in the case where the codomain is an m-dimensional space and by Bento et al. [SIAM J Optim, 28 (2018), pp. 1104-1120] for nonconvex finite-dimensional problems in terms of Clarke’s generalized gradients.