Cashless transactions via credit cards and e-wallets are becoming very common due to the ease of payment while purchasing goods or services. Proportional to online transactions, fraudulent activities are also increasing. Fraud in online transactions is becoming a major concern for customers and financial institutes. The traditional method such as manual review of transactions is both time-consuming and incompetent in detecting fraud on time. Developing an enhanced and efficient model using machine learning techniques can detect fraud with more efficacy. Implementing systems to detect fraudulent activities seems nearly impossible as many challenges bother the fraud detection systems such as imbalanced data, concept drift, generation of huge amounts of data in a short period, and many more. ML techniques use past transactions to learn the pattern and identify fraudulent activities. Online transactions benefit the industry and provide a hotspot for intruders to infiltrate the network and commit fraud. There are different areas where online financial fraud is widespread. This paper provided a comparative review of existing fraud detection models built using machine learning technologies in the area of online financial transactions and their novel contribution to the field. It also provides a comparison of the results using their performance matrices. The compared results show that the CNN with ADASYN data sampling technique outperforms in accuracy, precision, recall, and F1 score. The proposed hybrid model (Cu-DNNGRU + Self-Attention + DenseNet121) exhibits 99.95% accuracy, more than the accuracy exhibited by CNN + ADASYN.