Auricular deformities are one of the most common concerns in neonates and, if left untreated, can result in both psychological distress and functional complications. Timely diagnosis is essential for appropriate interventions which is proven to effective but must be done within the first few weeks after birth. However, limited clinical expertise and the intricate structure of the auricle pose challenges to consistent and early identification. This study presents a robust deep learning-based framework for the accurate detection and classification of newborn auricular deformities using high-resolution images from the BabyEar4k dataset, comprising 3,852 annotated baby ear images. The proposed pipeline follows a two-stage architecture: auricle localization using a Faster R-CNN model with ResNet-50 backbone, followed by cropping of the proposed regions to isolate auricular structures. These cropped regions undergo targeted preprocessing—such as contrast enhancement, sharpening, and Gaussian filtering to enhance auricular cartilage structure and mitigate the variations in image quality factors. The pre-processed regions are then fed into a classification module based on a Vision Transformer (ViT) with a custom head. For model interpretability, Randomized input sampling for explanation (RISE) is employed to generate saliency maps, highlighting the critical regions the model concentrated on while classifying the auricle. The proposed framework attained a mean average precision (mAP50) of 0.84 for auricle detection and achieved a classification accuracy of 84% using ViT classification module. The results reported are solely using the not-so-good quality images having illumination variations, inconsistent focus and occlusion. Saliency maps generated through RISE confirmed that the model consistently focused on relevant auricular substructures.