Toward Generating a Large Scale Intrusion Detection Dataset and Intruders Behavioral Profiling Using Network and Transportation Layers Traffic Flow Analyzer (NTLFlowLyzer)
摘要
In today’s digital landscape, network security and intrusion detection systems are crucial due to our growing dependence on interconnected systems and data exchange. IDS continuously monitors network traffic to detect threats and ensure the security and integrity of modern digital infrastructure. However, IDSs face several challenges, including low accuracy, high false positive rates, the absence of an effective behavioral profiling model, and the requirement for enhanced visualization capabilities. This paper introduces a groundbreaking pattern extraction and profiling system that addresses limitations in characterizing diverse network activities. We introduce a novel attribute selection algorithm, a groundbreaking approach for characterizing network activities, and a novel concept of local and global profiling, featuring the concept of a “super feature”. Our approach, which includes Attribute Extraction, Relation Extraction, and Entity Extraction, forms a robust foundation for precise activity characterization and accurate profiling. By emphasizing sub-behaviors through Local and Global profiling, we effectively mitigate the common issue of high false positive rates seen in previous methods. The approach culminates in the weighting of sub-profiles and the influence of the global profile on shaping comprehensive activity profiles, achieved through a neural network architecture. We perform practical implementation and validation by developing a new network traffic analyzer, NTLFlowLyzer, with an extensive set of over 300 features and introducing the updated benchmark data set BCCC-CSE-CIC-IDS2018. The experimental results showed that the proposed Local and Global profiling was effective in profiling different malicious activities.