Railway systems are an essential backbone of our society, facilitating transporting people and large quantities of goods across long distances. As the shift from cars to trains gains momentum, their reliability becomes even more essential. To ensure the efficiency of this transportation system, it is necessary to plan and execute regular inspection measures. In Germany, this testing is typically carried out using Ultrasonic sensor data from probes mounted on inspection trains. The research presented in this paper is based on collaborative work between partners from different research fields and inspection practitioners. The aim is to leverage real-world Ultrasonic field data from inspection runs and augment them with simulated rail defects that would otherwise occur too infrequently to enable meaningful Deep Learning analysis. Given the spatial and temporal characteristics of the data, modeling approaches that can effectively capture both elements by employing convolutional and recurrent neural network layers are compared. Combining these two layers offers a suitable solution for identifying and classifying defects in typical sensor data from routine inspections. Evaluation of the Model usability evaluation happens on indicators reflecting the practical usability of such. Every approach will be validated against essential metrics, such as AUC in binary prediction or True and False Positive Rate for classification. Considerations for the Probability of Detection (POD) are also made, as this metric is essential to evaluate Non-Destructive Testing methods.