This paper presents results from testing of machine learning techniques applied to drive-by bridge monitoring. Two different approaches using deep learning are presented to allow (i) the presence of damage to be detected and (ii) damage characteristics to be classified. Results from laboratory experiments show that an Artificial Neural Network, trained with the measurements from a passing vehicle, can be used to identify when cracking is present in the bridge beams. Results also demonstrate that a Convolutional Neural Network, trained with inputs from a calibrated finite element model, can be used to accurately classify the bridge condition and to estimate the number of cracks. It is also possible to estimate the location of the cracks, albeit with a lower level of accuracy for the least severe damage cases. The results of this study illustrate how deep learning techniques can be leveraged to enhance the damage detection and classification capabilities of drive-by bridge monitoring.