Pneumonia and Tuberculosis are two major illnesses that pose significant threats to global health. Effective intervention and management of these disorders require an efficient early diagnostic system. In our work, we introduce a feature fusion network for categorizing Chest X-Rays into Pneumonia and Tuberculosis. Our feature fusion network combines the output features obtained from the local feature extractor and global feature extractor. It can help the model to identify both global perspectives and fine details during classification. Global level extraction module extracts broader features and patterns across the entire chest region. Also, by incorporating a local level extraction module, our fusion model can identify subtle variations within certain lung areas, which may be overlooked by the global level extractor due to its tendency to focus on broader patterns. Additionally, we integrated numerous small-sized datasets to develop an adequate dataset, which we used to evaluate the suggested strategy. Our suggested approach takes advantage of modern deep learning strategies and attains a tremendous accuracy of 98.56%, surpassing several pretrained architectures. It illustrates how effectively the model can identify different classes, thereby defining the effectiveness of the proposed technique in medical image classification tasks. Thus, the pro-posed method may aid medical professionals in accurately diagnosing Tuberculosis and Pneumonia using Chest X-Rays.