Eye diseases can lead to irreversible vision loss if not detected early, highlighting the potential of automated diagnosis systems to assist health experts in timely detection. While recent deep learning models show high diagnostic performance, they often lack efficiency and rely on costly multi-modality inputs, limiting their practical deployment. We present QD-RetNet, a Quantized Distillation Retina Network that revolutionizes retinal disease classification by delivering clinical-grade diagnosis in a low-resource setting. Unlike conventional approaches that rely on paired multi-modal imaging data for better accuracy, QD-RetNet processes OCT and fundus images independently within a shared knowledge distillation framework, removing the need for large, paired datasets. Using Quantization-Aware Training (QAT), our model achieves 4 \(\times \) compression while retaining diagnostic accuracy close to that of larger, high-compute models. Exhaustive evaluation on TOPCON-MM, MMC-AMD, and MultiEYE benchmark datasets confirms our model’s robustness across a broad spectrum of real-world retinal disease prediction tasks. Code is available at https://github.com/ashutoshkr45/QD-RetNet .