The advances in technology has assisted the on-site inspection of historical masonry structures, which can be cost-efficient, effective and resource saving. This research presents an efficient and automated damage detection algorithm for heritage masonry structures using Faster Region Convolutional Neural Networks (FRCNN). It addresses the lack of public datasets by creating a labeled dataset for training, which at the moment is not publicly available for masonry structures. Moreover, the proposed FRCNN-based system efficiently detects and localizes damages, demonstrating promising speed and accuracy, making it a cost-effective solution for reducing subjectivity in inspection processes. The findings of this research can help in detecting the damages in the masonry structures with enhanced computational speed in a cost-effective manner.