Epilepsy is a neurological disorder characterized by recurrent, unprovoked seizures, and early diagnosis is crucial for effective management and treatment. However, the diagnosis of epilepsy, particularly in its early stages, remains challenging due to the subtle nature of seizures and the complexity of brain activity patterns. In this paper, we introduce the Medical-Informed Vision Transformer (MIVT), a deep learning architecture specifically designed to improve early epilepsy diagnosis from multimodal neuroimaging data. Our model integrates insights from both medical knowledge and state-of-the-art Vision Transformers (ViTs) to enhance the accuracy and interpretability of seizure detection and localization. The MIVT leverages the rich spatial and temporal features of Electroencephalography (EEG), enabling the system to learn discriminative features that correspond to early seizure precursors and biomarkers. We demonstrate the effectiveness of the MIVT on a large, multi-modal epilepsy dataset, showing superior performance over conventional deep learning models i.e., Inception V3, ResNet-50, and AlexNet by a margin of 17%. Our results indicate that the MIVT model outperforms existing techniques with a 93.55% diagnostic accuracy, 88.89% specificity, 98.72% AUC, 86.67% precision, and 100% recall. It shows potential in bridging the gap between machine learning models and clinical practice.