<p>Skin cancer is the most common cancer in the white population and among the most treatable when diagnosed early. Artificial intelligence (AI) has shown high diagnostic accuracy for skin cancer. Patient metadata has proven helpful for automatic skin lesion classification with AI. However, there is a lack of in-depth analysis on which metadata are most relevant. This study investigates the impact of patient metadata on skin lesion classification. We train 17 deep learning models using three fusion methods on two publicly available datasets: PADUFES20, composed of clinical images, and ISIC2019, composed of dermoscopic images. Additionally, we train models with different metadata subsets to evaluate their importance. For PADUFES20, models using images and metadata achieved 10.43% higher balanced accuracy than the baseline, and for ISIC2019, the improvement was 2.22%. We conclude that fusing images with metadata outperforms models using images or metadata alone. The results highlight that patient age in ISIC2019 and lesion-specific metadata in PADUFES20 are the most impactful. For future research, we make our model implementation a flexible tool for fusing images and metadata in various tasks.</p>