In intensive care units (ICUs), continuous monitoring of patient vitals is crucial for timely interventions and optimal outcomes. While traditional methods rely on direct observation by healthcare professionals, technological advancements have led to a growing interest in image-based monitoring systems, particularly utilizing closed-circuit television (CCTV) cameras within ICU environments. However, integrating such systems poses challenges, notably in extracting vital information from monitor images efficiently. Current approaches, including manual interpretation and specialized algorithms, are often laborious and error-prone. In response, this paper presents a robust pipeline for automatic extraction of patient vitals from ICU monitor images. Our approach involves automatic detection and segmentation of monitor screens, followed by extraction of relevant vital signs. By enhancing existing ICU environments with this technology, we aim to improve patient care and resource utilization. Our contributions include a powerful pre-processing pipeline, a comprehensive study of optical character recognition (OCR) frameworks, and a geometry-based heuristic implemented in kornia for vital sign detection. Through this work, we lay the groundwork for innovative solutions that can revolutionize patient monitoring in ICU settings. The method has been tested on a large dataset comprises with 10K ICU monitor images.