Civil infrastructure materials deteriorate over time due to various physical and chemical processes, such as corrosion and thermal cycles. Surface cracks on concrete structures are often the initial indicator of degradation, making it essential to assess their nature, extent, and dimensions to determine the current state of the structure and take preventive measures to avoid future damage. However, traditional manual and visual inspection methods can be labor-intensive, hazardous, and subject to subjectivity and error, making it challenging to track the evolution of defects between inspections. Deep learning convolutional neural network (CNN) algorithms, such as Mask R-CNN, provide a high level of abstraction and generalization that is achieved through training on extensively annotated datasets. These algorithms have the potential to accurately segment cracks in images, thus providing a reliable and efficient means of assessing the state of concrete structures. However, publicly available crack datasets used for training the network often contain idealized crack images without real-world noise, such as dirt, leakages, or graffiti. Our study demonstrates that adding real-world features to a dataset and augmenting the amount of data samples can improve the accuracy of the network. We utilize the publicly available CRACK500 dataset and modify it by eliminating low-quality samples and adding real graffiti patterns. By doing this, the network was able to differentiate between cracks from graffiti and wet concrete marks on real-world images, achieving a precision of 97.8% and a recall of 53.1%.