Lateral flow tests are widespread tools for the diagnosis of severe diseases, such as coronavirus, streptococcus, human immunodeficiency virus and hepatitis, and for drug abuse detection. The visual inspection of these tests is generally performed by skilled human operators. Despite their expertise, factors like subjectivity, physical conditions (e.g., illness, fatigue, stress) or light conditions can influence the analysis of the tests, increasing the possibility to generate errors in the evaluation. To overcome such issues, there has been a notable spread of Artificial Intelligence techniques, which can examine big amounts of data with excellent performance. In this work, we compare state-of-the art Convolutional Neural Network and Vision Transformer models for the problem of detecting drug abuse in lateral flow tests. We created a dataset used to train models for the capability of discriminating between normal (i.e., negative) and abnormal (i.e., positive) tests. Our outcomes show that the considered models represent valid alternatives for the problem. Moreover, the explainability analysis reveals that some of the models manage to identify the critical parts of the images, although the amount of captured information is limited.