Automatic Optimization-Based Methods in Machine Learning: A Systematic Review

Machine Learning (ML) is a subfield of Artificial Intelligence (AI) that has been applied to various fields ranging from industrial to medical sectors, to perform miscellaneous Computer Vision tasks such as image classification, image segmentation, object detection, and language modeling. Notwithstanding, having a suitable model with practical applicability requires performing appropriate structural operations upon datasets, building adequate CNN architectures from the scratch or resorting to the ones available in the state-of-the-art, and, either way, parameterizing them to improve machine learning skills, usually, in a trial-and-error fashion. Aligned with this context, despite the several semi-/fully automatic approaches that can be found in the literature, (e.g., grid search for hyperparameter fine-tuning, auto-Machine Learning for self-configurable model development, and automatic methods for data arrangement and augmentation), which are often integrated with combination to establish automatic pipelines for the effective implementation of solutions powered by AI, surveys documenting such topic seem to be scarce. Therefore, the main goal of this work is to present an updated yet extensive literature review focusing on this class of approaches, considering the importance of their role in the perspective of ML Optimization.