Rare medical illnesses account for less than 5% of clinical presentations but often involve the threat to life and most probably get mislabeled since there is not sufficient labeled data. Traditional or non-traditional deep models have very little capacity to deal with such rare classes and require ginormous amounts of training data. Zero-Shot Learning (ZSL) might be the optimal option because it is able to predict across new conditions from semantic relations. Our results suggest a diagnostic imaging method using ZSL with visual-semantic embeddings to identify abnormal diseases from unseen classes for imaging modalities like MRI and CT scans. Clinical benchmark sets are tested with 78% top-1 accuracy on novel classes and a 12% gain over few-shot models. Our method achieves scalable generalization to novel diseases, potentially revolutionizing early disease diagnosis in under-explored clinical spaces.