Laryngeal cancer (LC) and other benign conditions are major concerns in modern ear, nose, and throat medicine. A comprehensive evaluation of the larynx should employ flexible or rigid endoscopes to identify early-stage lesions, possibly enhanced with advanced imaging techniques such as Narrow Band Imaging (NBI) to empower tissue visualization. Factors that make the detection, diagnosis, and treatment of LC challenging include the huge amount of uninformative frames and the expertise-dependent nature of the assessment, leading to time-consuming procedures with high cognitive loads and the possibility of missed detections and misdiagnoses, especially for less-experienced clinicians. Deep Learning (DL) approaches have recently been studied regarding frame quality assessment, abnormal mass identification, and their margins definition to improve diagnostic accuracy and surgical outcomes. In this work, we proposed the integration of several Convolutional Neural Networks (CNNs) into a single computer-aided system for the assistance of less-experienced otolaryngologists, by directing their attention toward good-quality frames from which lesions can be automatically detected and characterized. We addressed the following challenging tasks: informative frame selection, lesion detection, lesion classification, and lesion segmentation. The developed system demonstrated a good trade-off between efficacy metrics and real-time performance, and the potential for clinical applications.