The process of kinship verification through facial images poses significant challenges due to subtle variations within and between classes and gender bias influences. Traditional methodologies often face biases from gender-related features and struggle with overlapping decision boundaries in the metric space. In this paper, we propose an innovative framework combining Angular Margin Metric Learning (AMML) with Gender Relation Discriminating Adversarial Decoupling (GRDAD). The AMML component enhances feature discrimination by optimizing angular margins, while GRDAD mitigates gender biases through a gender label construction method and Adaptive Weighted Cross-Entropy (AWCE) loss. We rigorously evaluate our method against three benchmark datasets—KinFaceW-I, KinFaceW-II, and TSKinFace—and show competitive performance, surpassing existing methodologies in accuracy and interpretability. Our approach focuses on extracting kinship-relevant features and enhancing model robustness across various kinship relationships. Additionally, our framework can be seamlessly integrated into current systems to effectively decouple gender information, thus augmenting robustness in diverse kinship relations.