With the rapid development of the construction industry, society's requirements for the appearance quality of buildings are increasing. In this context, traditional building surface defect detection methods have exposed many shortcomings, such as low efficiency and results that are easily interfered by subjective factors, which makes it difficult to meet the high standards of quality inspection for modern large-scale construction projects. This research conducts a comprehensive examination of the present utilization of machine vision technology in the automatic detection of building surface defects. It also presents an innovative methodology that combines reflective fringe deflectometry with deep learning. The proposed approach commences with the utilization of reflective fringe deflectometry to record the three-dimensional morphological data of the building surface, facilitating the initial location of defect zones. Subsequently, the three-dimensional data is pre-processed by means of a deep learning algorithm to accurately extract potential defect regions. This method not only considerably diminishes the volume of data to be processed but also boosts the accuracy of defect identification. Experimental outcomes show that the detection model developed in this study attains an accuracy rate of over 95% in identifying cracks within intricate building surface environments. This remarkable performance is mainly ascribed to the Convolutional Neural Network (CNN) model's capacity to deeply learn and recognize crack patterns through extensive training with a large dataset of labeled samples.