Traditional Intrusion Detection Systems (IDS) frequently rely on anomaly and signature-based methods, which have limits in terms of accuracy, scalability, and flexibility. Anomaly-based systems generate a high number of false positives, causing operational problems, but signature-based solutions cannot protect against zero-day assaults. To overcome these problems, the proposed system improves IDS performance in dispersed data engineering contexts by utilizing Long Short-Term Memory (LSTM) networks, which are a type of Recurrent Neural Network (RNN). Furthermore, network data has temporal correlations, which the LSTM-based IDS can utilize to reduce false positives and improve detection accuracy. The methodology ensures maximum fault tolerance and resiliency while scaling to handle terabytes of real-time data by distributing the LSTM model across a cluster and cloud-based architecture. Research results reveal that the proposed approach outperforms existing IDS approaches. For instance, the KDD Cup 99 dataset, achieved 92.5% accuracy, which is higher than other top algorithms in this scenario defined as: Precision (88.4%), and recall (91.2%). F1 score: 89.7%. However, the scores acquired from the NSL-KDD dataset were 91.2% F1 score and 93.2% accuracy, which is rather low in comparison to our results but can provide us with an elementary understanding of the outliers present in both approaches. Furthermore, the techniques greatly reduced false positives and false negatives by overcoming modern cybersecurity challenges.