COVID-19 is an infectious respiratory disease that has impacted many people’s lives. Using computer-assisted solutions is crucial for the early detection and treatment of infected patients. Computer vision and deep learning techniques can accelerate the process of early identification of infection. Computer vision algorithms aid in medical imaging data analysis for characteristic patterns associated with COVID-19. Deep learning models learn complex patterns and features from large-scale datasets; thus, bearing good applicability in medical image analysis and diagnosis. In this paper, we propose a deep learning model—C19XNet, capable of detecting and classifying the presence of COVID-19 after processing the chest X-ray images. In the given study, compared to existing models such as VGG16, ResNet50, Dense-Net121, MobileNetV2, and SqueezeNet, by preprocessing images through non-local means (NLM) denoising and optimizing the deep learning model using batch normalization and weighted dropouts C19XNet demonstrated the highest accuracy of 93.62 and F1 score of 0.94. This is achieved through an optimized network architecture with three times reduction in training time; thus providing an automated classification method that eases diagnosis in resource constrained settings.