Challenges and Opportunities for Network Intrusion Detection in a Big Data Environment

Advanced network sensors, data storage, and processing technologies allow the accumulation of logs, network flows, and system events from various sources in terabytes of heterogeneous data. The abundance of data can be used to train and validate multiple machine learning approaches and algorithms to detect anomalies and classify network attacks. This paper presents the state of the art in data preprocessing, feature selection, and applying various machine learning methods for intrusion detection. It outlines the main challenges in big data analytics, the functional requirements to related tools and applications, and the opportunities provided by combining the outputs of several methods to increase the accuracy of detection and decrease the number of false alarms. Finally, the authors propose an architecture of an intrusion detection system combining offline machine learning and dynamic processing of data streams.