Disease segmentation from X-ray images is important in medical diagnosis and treatment planning. Timely and accurate detection of different diseases from X-ray images can help clinicians diagnose patients accurately within the shortest possible time, improving patient outcomes. Conventional approaches to the segmentation of diseases are feature engineering dependent and need considerable data pre-processing, restricting their capability and effectiveness. In this work, we present diseases automatically on vision transformer, using MAR architecture to segment diseases automatically from X-ray images. ViT was originally designed for a natural language task, but recently, it has been shown that the architecture performs well on image recognition tasks. Methods: We employ a multi-model approach where we build upon the strengths of various ViT-based models to obtain superior performance. Our model also employs self-attention mechanisms, enabling the models to focus on relevant areas in the big images. We tested our method on a large-scale dataset of X-ray images of multiple diseases, including pneumonia detection, tuberculosis detection, and lung cancer detection. Our results demonstrated a better accuracy and computation time than the state of the art. Our method was robust against changes in lighting conditions, image quality, and various diseases. In our proposed method, automated disease segmentation based on a multi-model from ViT architecture can help convert the speed and accuracy of X-ray image diagnosis into clinical practice. Our results demonstrated a mean dice similarity coefficient of 0.896 and an intersection over union score of 0.844, surpassing state-of-the-art methods in accuracy and computational efficiency.