This research presents DermaVLM, an innovative Vision-Language Model designed to transform skin disease diagnosis by combining computer vision and natural language processing techniques. The approach integrates Convolutional Neural Networks (CNN) for image-based classification and Retrieval Augmented Generation (RAG) for contextual insights, mirroring the diagnostic process of experienced dermatologists. Our technical foundation follows a three-step training protocol that optimizes visual feature extraction and knowledge integration, starting with ResNet101-based CNN architecture for comprehensive dermatological condition identification. The Llama 3.2 11B Vision model follows the CNN to provide language understanding and is fine-tuned for dermatological terminology and concepts. A domain-specific adaption is incorporated where the model is fine-tuned on dermatology-specific datasets, including SkinCAP and SCIN. On comparing the optimized model, the performance metrics indicate a 92% classification accuracy and a 47.9% reduction in inference time; token efficiency is improved by 73.9%, and a resultant 44.5% increase in semantic alignment with medical images as compared to an unoptimized baseline, enhancing the model’s utility in real-world applications. Based on this approach, we aim to provide DermaVLM as a scalable, efficient, and clinically reliable solution to improve dermatological accessibility in underserved communities by bridging the gap between advanced medical AI and communities with scarce medical expertise.