Medical Visual Question Answering (MedVQA) aims to develop models to answer clinically relevant questions on medical images. A major challenge in developing VQA for the Medical domain is the unavailability of large, well-annotated MedVQA datasets. Using transfer learning from ImageNet and finetuning on the MedVQA dataset is not helpful as visual concepts of ImageNet images and medical images differ. Therefore, this paper focuses on the problem of the lack of a large MedVQA dataset by employing a novel pre-training technique for the visual encoder. Our pre-training framework uses contrastive and restorative learning to learn fine-grained semantic representations from large, unlabelled medical images available online. We finetune our pre-trained visual encoder on the MedVQA dataset. Our experiments show that the combination of contrastive and restorative learning significantly improves the performance of MedVQA systems. We evaluate our model on three MedVQA datasets. The source code is available at https://github.com/Vasudha27/CRP-for-MedVQA .