The paper focuses on the possibility of using transformer-based and generative models for mining imbalanced datasets. We briefly review recent developments in constructing transformers and generative models and their application in machine learning. Next, we explain how to use public domain libraries to build a local transformer-based classifier, that can be used solo or in conjunction with a combined transformer-based and VAE model for generating synthetic minority data or generative model for synthetic minority data generation. The approach is validated in the computational experiment showing that transformer-based classifiers perform well and can be an interesting alternative to classic methods of mining imbalanced datasets.