Accurate disease classification from chest X-ray images remains a significant challenge in medical imaging, particularly when multiple diseases coexist or exhibit similar visual features. In this paper, we propose TagVisionNet, a framework designed to enhance disease classification by integrating diagnostic label information with chest X-ray images. Unlike conventional methods that rely solely on global image features, TagVisionNet incorporates disease labels as additional inputs, enabling the model to simultaneously leverage both image data and label semantics. By computing the inner product between the feature representations of disease labels and image features, the framework adopts a similarity-based classification approach. This design allows the model to capture semantic relationships between lesion features and their associated labels, thereby improving classification accuracy, especially in cases involving coexisting or visually similar diseases. We evaluate the performance of the proposed method on the Chest X-ray14 dataset, a widely recognized benchmark for chest X-ray image classification. Experimental results indicate that TagVisionNet outperforms traditional models, including ResNet and Vision Transformer. These findings underscore the effectiveness of incorporating label information to achieve more accurate and robust predictions in medical image analysis.