Speeding up Line Search for Composite Objective Function with a Linear Inside Part

The paper illustrates a significant acceleration of the line search procedure for composite objective functions with a linear inside part. The memoization technique is used, and our implementation of the line search primarily employs directional derivatives instead of gradients. The domain of application of the paper results covers various penalty functions of practical interest. In experiments, we consider symmetric matrix games and two-layer neural networks. Numerical experiments predominantly employ the L-BFGS algorithm, leveraging our custom implementation. A series of tests showcase the effectiveness of this implementation. The rationale behind memoization is elucidated, alongside potential avenues for future research. The results of the experiments validate the efficiency of the proposed approach. Our approach is competitive with other methods for solving symmetric games on a certain class of matrices. In multi-layer neural networks, significant acceleration is observed in computing function values along fixed rays