Indirect Structural Health Monitoring of bridges using a passing vehicle is gaining momentum due to its notable advantages such as single/minimum sensor requirement, economic over installation and maintenance of sensing systems, fast diagnosis, convenience, non-closure of traffic, and scalability. Much research is underway to identify the modal parameters and damage using a drive-by vehicle on a bridge. Addressing the challenges, such as extracting the increased number of modal frequencies for damage detection, minor damage detection, handling environmental/operational variability, and measurement noise, are paramount. Also, with the advent of robust sensing/data acquisition systems, enormous data is collected from the structure, for which the use of machine learning techniques for SHM is being sought to a large extent. However, many times, the availability of enough training data under various real scenarios is not practically possible. So, there is a need to minimize the requirement for large training data. In the context of bridges, this paper presents a technique that classifies the state of health of a group of bridges under variability. The technique is validated with numerical studies using a group of simply supported beams with varied lengths, carrying a moving load, and found that it can classify the group of bridges under two classes- Healthy and Damage with high accuracy.