Robust Optimization of Discontinuous Loss Functions

Discontinuous loss functions are prevalent in engineering, sciences, data science, and machine learning applications. In engineering and the sciences, loss or constraint functions that require differential or partial differential equations to be solved often manifest discontinuities because of adaptive time stepping or remeshing. This results in abrupt changes in the discretization error, resulting in discontinuities. In the data sciences, loss functions computed from batches with different samples are discontinuous. These discontinuities are the result of abrupt changes in the sampling error. Alternatively, the loss and activation functions can be the source of discontinuities. This chapter gives illustrative examples of some of the origins of discontinuous loss functions and some basic strategies for exploiting gradients to optimize loss functions induced with discretization and sampling errors, i.e., gradient-only optimization.