AI-related technologies based on machine learning have been rapidly developing in recent years and still advancing. On the other hand, misrecognition and/or biased outputs of AIs caused by a class imbalance problem in their training dataset would become a serious social issue. This is the case with not only labeled datasets but also unlabeled ones since the class imbalance could be implicitly contained even in unlabeled datasets. So far, a lot of methods for solving the class imbalance problem have been proposed, none of which focuses on unlabeled datasets. Hence, in this paper, we propose a method for reducing the implicit class imbalance in an unlabeled dataset. We call this task unlabeled dataset detoxification, regarding the implicit class imbalance as a kind of “toxin” for unlabeled datasets. In the proposed method, we first train a GAN using a target imbalanced dataset itself and generate a lot of new data samples using it. In this process, we estimate the rarity of each generated sample based on the pseudo-Jacobian of the trained GAN generator. After that, we add only the GAN-generated samples with high rarity into the target dataset. This allows us to reduce the implicit class imbalance. We conducted experiments on an unlabeled face image dataset that is imbalanced in terms of race, whose results demonstrate the effectiveness of the proposed method; it can effectively generate rare-class samples that are helpful in boosting face recognition accuracy for identities of the rare classes, namely rare races.