With the advancement of China’s rapid urbanization and the need for the protection of historical buildings, durability and life prediction methods for buildings have attracted extensive attention from scholars, especially the deformation detection of the overall structure of the building and the detection of tilt rate and settlement. To this end, this paper proposes a visual recognition-based damage detection and life prediction method for buildings, which utilizes a dedicated image acquisition device to geometrically analyze and calculate the crack images of building walls, especially for load-bearing beams and load-bearing walls. A life prediction model is established for the degree of cracking of the wall structure of a building from an image perspective, and the deformation of load-bearing walls and beams is comprehensively analyzed in conjunction with the structural mechanics of the building, on the basis of which the damage trend in the later stages of the building is predicted. After the sample images collected in the field, combined with the principles of architecture, the proposed prediction method is compared with the theoretical calculation and visual prediction, and the comparison results show that: the proposed visual recognition method for the prediction of crack development in buildings is of scientific significance for guidance and basically conforms to the laws of the mechanics of the building structure, and it is a kind of non-contact, low-cost, and high-efficiency prediction method for the life span of the building, and it has a certain value of engineering application.