The widespread use of credit cards has introduced significant security challenges, urging financial institutions to adopt more robust fraud detection mechanisms. Traditional models often struggle with imbalanced and small datasets, which reduces their effectiveness in identifying fraudulent transactions. This paper proposes a novel fraud detection model integrating the TabTransformer with cost-sensitive learning techniques. The TabTransformer captures complex interactions within tabular data, while cost-sensitive learning increases the model’s sensitivity to fraudulent transactions by assigning a higher penalty to misclassifications of fraud cases. Our model is evaluated on three recognized datasets and outperforms state-of-the-art models, achieving superior accuracy, precision, recall, and F1-score results, thus significantly reducing false positives and negatives. Specifically, it achieved an accuracy of 99.94%, a precision of 99.89%, a recall of 100%, and an F1-Score of 99.94%.