During driving, the driver’s attention may be away from the road, commonly known as distracted driving. This is a key reason leading to traffic accidents. In the literature, most studies proposed machine learning algorithms for driver distraction recognition to classify the latest driving status. However, applying these algorithms cannot fully benefit the prevention of road traffic accidents and, thus, the reduction of deaths and injuries. In this paper, the driver distraction problem is formulated to predict future behaviors (distracted driving events or proper driving). The research effort is devoted to the feature extraction of drivers’ images using a deep kernelized autoencoder. Various common kernel functions are analyzed, including linear, polynomial, sigmoid, Gaussian, Laplacian kernel, radial basis function, and ANOVA radial basis function kernels. Performance evaluation of our work reveals F1-score ranges of 48.9–86.8% and 50.2–87.1% using two hidden layers and three hidden layers, respectively. Our work outperforms two existing works by an average of 3.52% in 1–5 s prediction intervals and 3.76% in 1–3 s. Various future research directions are also discussed.