In this work we propose a general nonmonotone line-search method for nonconvex multiobjective optimization problems with convex constraints. At the kth iteration, the degree of nonmonotonicity is controlled by a vector \(\nu _{k}\) with nonnegative components. Different choices for \(\nu _{k}\) lead to different nonmonotone step-size rules. Assuming that the sequence \(\left\{ \nu _{k}\right\} _{k\ge 0}\) is summable, and that the ith objective function has Hölder continuous gradient with smoothness parameter \(\theta _i \in (0,1]\) , we show that the proposed method takes no more than \(\mathcal {O}\left( \epsilon ^{-\left( 1+\frac{1}{\theta _{\min }}\right) }\right) \) iterations to find a \(\epsilon \) -approximate Pareto critical point for a problem with m objectives and \(\theta _{\min }= \min _{i=1,\dots , m} \{\theta _i\}\) . In particular, this complexity bound applies to the methods proposed by Drummond and Iusem (Comput Optim Appl 28:5–29, 2004), by Fazzio and Schuverdt (Optim Lett 13:1365–1379, 2019), and by Mita, Fukuda and Yamashita (J Glob Optim 75:63–90, 2019). The generality of our approach also allows the development of new methods for multiobjective optimization. As an example, we propose a new nonmonotone step-size rule inspired by the Metropolis criterion. Preliminary numerical results illustrate the benefit of nonmonotone line searches and suggest that our new rule is particularly suitable for multiobjective problems in which at least one of the objectives has many non-global local minimizers.
