Railway accidents are getting increased in recent days because of the poor maintenance in the railway tracks. Most of the accidents happening in the past days. Are due to the railway track cracks and a normally objects present in the railway tracks. The presented system is focused on detecting the anomaly objects present in the railway tracks by using deep learning algorithm with obstacle tracking model. Various challenges. Are faced with the limitation in the training data set. The presented system is focused on live tracking of railway tracks with SSD mobile net (single shot detector model) associated with VGG16 architecture to comprehensively extend the detection and segmentation of anomaly object present in the railway track. To enhance accuracy, early warnings, and overall railway track safety the need for detection of foreign objects in the tracks are important. The presented system considers obstacle detection through ultrasound sensor, communication through NodeMCU enabled customized cloud using Internet of things (IoT) is developed. The presented approach is tested with real time environment and evaluated.