Lung diseases affect millions of people worldwide, posing a serious health challenge. Timely and accurate diagnosis is crucial for improving patient outcomes and ensuring effective treatment. In this paper, we propose a multi-modal model that integrates both chest X-ray images and clinical information text to improve the efficiency of lung disease classification. Our proposed model trains a Support Vector Machine (SVM) model on top of the fine-tuned VGG16 model and Extreme Gradient Boosting (XGBoost) model with TF-IDF feature extraction. We started by collecting a new real-world dataset of chest x-ray images and clinical information at General Hospital of An Giang area province. Experimental results on newly collected real dataset show that the VGG16 model, which uses chest X-ray images, achieves an accuracy of 62.66%, indicating limitations when relying solely on image data for lung disease classification. With clinical text data,he SVM and XGBoost models with TF-IDF feature extraction achieves accuracy of 87.40% and 89.26%, respectively. Our multi-modal fusion model achieves the highest accuracy of 89.64%. These results highlight the effectiveness of our multi-modal approach, providing a more accurate and comprehensive diagnosis of lung disease classification.