<p>Recent developments in the railway domain have outlined the need to design reliable automatic monitoring systems, whose applicability is strongly dependent on the capability to timely detect potential obstacles on the tracks, including unusual ones such as rocks: even if typical of the railway environment, rocks are still challenging to recognize with traditional object detectors. In this paper, we propose a two-stage approach for the recognition of rocks on railway tracks, in which the region of interest, identified through a semantic segmentation network, is divided into patches analyzed by a CNN-based binary classifier, without sacrificing localization capability. To achieve a performance assessment aligned with real application requirements, the proposed system was tested on both static images and videos collected by a camera mounted on board the train, to which synthetically generated rocks were added, modifying their size and position in order to follow the perspective of the train’s movement. The experimentation conducted, over both still images and videos, confirm the effectiveness of the proposed approach, which confirms to be a viable solutions for real time rocks detection in real scenarios.</p>