The location and size of concrete cracks are critical parameters for structural health monitoring and safety evaluation. In this work, a new quantitative method based on Convolutional Neural Networks (CNNs) is proposed to automatic extract the length, width and location of cracks in concrete beam. First, many images of surface crack from reinforced concrete beam under different loadings are collected, and then a database for surface cracks for reinforced concrete structure is established using a sliding window. Second, a segmentation model based on the Deeplab V3 + network is proposed, which can quickly recognize and segment the surface cracks in concrete beam. Three evaluation metrics, such as ACC, the F1-score, and IoU are adopted to evaluate the efficiency and accurate of the proposed model, and they were reached to 99.1%, 82.1%, and 71.2%, respectively. Meanwhile, the proposed model exhibits better ability than other traditional models. Moreover, Zhang Parallel Refinement Algorithm is applied to connect of cracks, and the width, length, and position of each whole crack are extracted and validated by the experimental results. This means the proposed method can extracts the cracks information on concrete structures with high efficiency and accuracy.