AI-Driven Big Data Quality Improvement for Efficient Threat Detection in Agricultural IoT Systems

Data is one of the most valuable resources an organization may have; it can have a significant influence on its long-term performance, or even its existence. With the ever-increasing volume of data generated and collected every day from the Internet of Things (IoT), social media, and other sources, big data quality and security have become two of the most crucial concerns confronting organizations, especially in the field of agriculture, where IoT devices are increasingly being used to collect and monitor data on soil conditions, nutrient deficiencies… On the one hand, data must be carefully protected to prevent attacks or violations of its confidentiality, integrity, and availability. On the other hand, it must be of good quality for an efficient and effective decision-making process. This comprehensive study aims to explore both Big Data Quality and Big Data Security, as well as the potential conflict between them. In our experiments, we employed the CICIoT 2023 dataset, a novel and comprehensive IoT attack dataset that has never been used before. We appropriately preprocessed the dataset and applied different algorithms to correctly classify the traffic. We then present our proposed AI-based approach to improving big data quality to enhance security. We also demonstrate that our approach improves the accuracy, generalizability, and reliability of the data, resulting in high-quality data that we properly fed into our algorithms for accurate threat detection. Hence, this work allowed us to delve into various Machine Learning and Deep Learning algorithms, namely, K-nearest neighbors (K-NN), long short-term memory networks (LSTM), deep neural networks (DNN), and ensemble methods. As a result, when compared to other research, our approach achieved better threat detection.