The Curse of Training and the Blessing of High Dimensionality

This chapter presents the main technical challenges to a rigorous engineering discipline that automates the scientific process. The number one challenge to what is presented in this book and many other theories is the uncertainty introduced by training. The fact that training does not guarantee to converge to a global minimum makes us look at engineering rigor through a blurry lens. Connected to it is the second challenge, which is also an opportunity: the behavior of models in high dimensionality. Reproducibility, which is described in Chap. 15 , would be trivial to implement if training was guaranteed to converge to a global optimum every time. Explainability (see Chap. 14 ) would be much easier if a straight reduction of information from the input to output, that is, a straight reduction into lower dimensional spaces, could always be the solution. However, modeling with higher dimensional spaces, that is, the introduction of virtual dimensions, allows to create so-called embeddings that solve problems that are otherwise ill-posed (see Definition 2.8 and the discussion in Sects. 11.3.3 and 9.4 ).