Ridge Function Machines

Ridge function machines implement a computationally tractable and theoretically sound means of creating approximations to functions given by data. A ridge function is a function \(\mathbb {R}^{n}\to \mathbb {R}\) given by \(\boldsymbol {x}\mapsto g(\boldsymbol {w}^{T}\boldsymbol {x})\) where \(\boldsymbol {w}\in \mathbb {R}^{n}\) and \(g\colon \mathbb {R}\to \mathbb {R}\) . A ridge function machine is a sum of ridge functions \(\sum _{j=1}^{m}g_{j}(\boldsymbol {w}_{j}^{T}\boldsymbol {x})\) . While it is computationally infeasible to allow for arbitrary continuous functions \(g_{j}\) in this sum, we can approximate such functions by means of linear combinations of B-splines, for example. Given data points \((\boldsymbol {x}_{i},y_{i})\) , \(i=1,2,\ldots ,m\) , the regularized ridge function approximation problem \(\min _{g\in \mathcal {G}}N^{-1}\sum _{i=1}^{N}(g(\boldsymbol {x}_{i})-y_{i})^{2}+\alpha \left \Vert g\right \Vert { }_{L^{2}}^{2}\) can be solved in \(\mathcal {O}(N+p)\) time where \(\mathcal {G}\) is the span of p uniformly spaced B-splines. For approximation by sums of ridge functions, a block Gauss–Seidel method can be used. The use of a limited number of weight vectors \(\boldsymbol {w}_{j}\) means that the method effectively performs an orthogonal projection onto a low or modest-dimensional subspace, an example of dimension reduction. Gradients with respect to the weight vectors \(\boldsymbol {w}_{j}\) can also be efficiently computed.