Recent advances in AI-based medical diagnosis have demonstrated impressive accuracy. However, concerns remain regarding the fairness of these models across demographic groups. Gender-related biases embedded in skin lesion images may compromise diagnostic equity and lead to systematic disparities in clinical decision making. In this study, we investigate gender-specific visual bias signals in skin lesion classification using the International Skin Imaging Collaboration 2019 dataset. We trained ConvNeXt models on male-only, female-only, and mixed-gender datasets, and evaluated them across all test sets. In addition, we selected 100 samples per signal from our dataset based on visual prominence scores across 10 dermoscopic features known to influence skin lesion appearance. Our results reveal systematic disparities in model performance across gender groups. For example, the Blue signal led to a sharp performance drop when the female model was evaluated on male data, while the male model performed substantially better on the same subset. This contrast highlights how certain visual signals can hinder cross-gender generalization. These findings suggest that certain visual features affect model reliability depending on patient gender, raising concerns for fairness in real-world clinical deployment. Our work provides empirical evidence and diagnostic insights that can support the development of bias-aware dermatological AI systems.