Pareto-Informed Multi-objective Neural Architecture Search

Aiming at the auto-design of powerful neural architectures with a requirement of compromising multiple objectives, this paper introduces a novel approach called Pareto-informed Multi-objective Neural Architecture Search (PiMO-NAS), which employs a solution generator influenced by Tchebycheff decomposition to explore the objective space of multi-objective NAS. Our methodology initiates with a transformation of discrete search space into continuous form, followed by iterative solution optimization, in which Gaussian Process (GP) surrogate models are utilized to establish a mapping from decision space to objective space. Subsequently, a solution generator, directed by preference vectors from the objective space, is designed to generate decision vectors to map the weights from the objective space back to the decision space. This solution generator is further optimized based on the gradients derived from the GP models. To ensure diversity in the solution pool, the solution generator synthesizes new candidate solutions guided by preference vectors generated by a well-designed adaptive sampler. In order to verify the performance of the proposed PiMO-NAS, a series of experiments were conducted within two typical NAS search spaces (i.e., the Once-For-All(OFA) and AutoFormer based ones, covering both convolutional neural networks and vision transformers), and more than 30 state-of-the-art NAS methods and models were employed for performance comparisons. Experimental results showcase that our approach can outperform most peers in terms of search time and solution quality, and has fantastic ability to efficiently discover high-performing neural architectures. In the OFA-based search spaces, compared with the MSuNAS, the proposed PiMO-NAS was able to achieve similar performance in two-thirds of the number of iterations, thereby saving about 24% of the search time. In the AutoFormer-based search space, we successfully approached a strong baseline formed by a single-objective evolutionary algorithm with restricted parameter quantities, approximating the entire Pareto front in a comparable timeframe.