Unpigmented skin cancer is the most prevalent form of cancer, and it burdens healthcare substantially even if it is not as aggressive as the more well-known malignant melanoma. Dermatoscopy images are commonly used for diagnosis, but differentiating between the many sub-diagnoses is a hard task. In this study we focus on these unpigmented cancers, performing both detection of basal cell carcinoma as well as fine-grained classification of its subclasses. We do this using a new dataset with more than 2’000 cases from a fair-skinned population. A deep learning model is specially designed for the task, handling pairs of polarised and non-polarised dermatoscopy images as input. We investigate transfer learning with different backbones as well as adding a mid-step of contrastive learning. The performance is compared to the accuracy of dermatologists on the subset of our test data where we have additional ground truth from histopathological diagnosis. This is the first study focusing on fine-grained classification of unpigmented lesions using only dermatoscopic images, and we reach a balanced accuracy of 39.4%, to be compared to 51.9% for dermatologists. This is a promising first step towards an algorithm to assist dermatologists in their work and we hope that this will open up for further studies on this interesting problem.