Mobile Internet technology and its widespread use in finance have led to the digitization of many financial businesses. Computers and mobile devices provide convenient financial services, but also generate significant amounts of unstructured behavioral data. Despite enormous efforts in customer security, losses from online fraud continue to rise. To overcome these drawbacks in this paper introduced a novel approach, Revolutionizing Fraud Detection in Real-Time Financial Transactions using AI-based Boundary-Integrated Neural Networks with Starfish Optimization Algorithm (FD-RTFT-BINN-SOA) is proposed. Initially, input data are collected from a Fraud Detection Transactions Dataset. Then the collected input data are preprocessed using Adaptive Tracking Dual Nested Kalman Filter (ATDNKF) is used to data cleaning and preparation. Then the classification is done by Boundary Integrated Neural Network (BINN) for effective real-time fraud detection in digital payment systems and classified such as not fraud, fraud. To enhance the classification accuracy, the Starfish Optimization Algorithm (SOA) is utilized to optimize the parameters of the BINN. Finally, the performance of proposed FD-RTFT-BINN-SOA method provides 27.15%, 26.09% and 28.10% higher accuracy and 29.03%, 25.23% and 29.1% higher recall. When comparing with existing techniques like Online payment fraud detection model utilizing machine learning techniques (OPFD-MLT), An intelligent payment card fraud detection system (IPC-FDS) and An effective fraud detection utilizing competitive swarm optimization driven deep neural network (EFD-CSO-DNN) methods respectively.