<p>Lithology classification based on well log data has attracted the attention of geologists, as the extraction of rock samples from the wellbore is a costly process. Although some machine learning techniques have recently been applied to the problem, the existing studies are not standardized in terms of data processing and evaluation protocols. This paper aims to bridge this gap by providing common ground for the comparison of prospective works on lithology classification from well logs. The proposed benchmark is inspired by a thorough review of the literature, from which the main aspects of an experimental protocol are captured under the principles of simplicity, variety, impartiality, and meaningfulness. The public datasets used in the experiments are the FORCE and Geolink repositories. Ten methods are tested, including six deep learning models and four shallow machine learning algorithms, and compared by nine distinct performance metrics under cross-validation. Different sequence lengths are tested as input for deep learning models to assess their contextual understanding capabilities. The experimental results using the proposed setup show that context-insensitive models outperform most context-sensitive ones, but with high correlation. The study also reveals that the problem of lithology classification is still not solved. Therefore, the benchmark presented herein serves as a fair protocol to assess different methods and can be used by future works as a simple and unbiased way to compare new approaches in lithology classification.</p>