Pneumonia remains a critical global health challenge, with an estimated 150 million new cases and approximately 2.5 million deaths annually. It is a predominant cause of mortality in pediatric and geriatric populations, often exacerbated by underlying chronic comorbidities and immunocompromised systems. The COVID-19 pandemic further intensified the incidence of viral pneumonia, underscoring the urgent need for advanced preventive and therapeutic interventions. In this study, we designed a modular framework for analyzing chest X-ray(CXR) images aimed specifically at detecting various pathogens of pneumonia. The segmentation module, implemented through a custom encoder-decoder architecture with attention mechanisms, effectively delineated regions of interest, particularly the lungs, enabling a detailed exploration of anatomical variations associated with pulmonary conditions. It achieved an accuracy, IOU and Dice coefficient of 99.08%, 96.49% and 98.21% respectively. The classification module focused on analyzing global patterns in the entire image, achieving high accuracy with binary classification reaching 98.77%. The precision, recall and F1-score for the proposed model stands at 98.80%, 98.74% and 98.76%. This modular approach, with its emphasis on task-specific optimization, offers a robust and flexible framework for advancing diagnostic capabilities in pneumonia detection and medical imaging.