Self-supervised Learning for Anomaly Detection in Large-Scale Cloud Infrastructure
摘要
Anomaly detection in large-scale cloud infrastructure is a necessary task that keeps system performance and security, and reliability in place. Rule based and supervised learning are unable to cope with the dynamic nature and large-scale challenges of the cloud environments because of the characteristics of the system. The purpose of this study is to present a self-supervised learning (SSL) framework of anomaly detection in cloud infrastructure by utilizing three state-of-the-art models: Bootstrap Your Own Latent (BYOL), Denoising Autoencoder (DAE), and Time series Generative Adversarial Network (TimeGAN). These are cloud performance metrics models (CPU utilization, memory usage, network traffic, disk I/O) that are used to analyze if there are anomalies in real time. To address this problem, the use of Amazon EC2 instances for data collection with data is achieved by a real-time streaming pipeline that uses Apache Kafka, Spark Streaming, and OpenTSDB. Experimental evaluations show that TimeGAN surpasses (93.5%) in detection accuracy, and BYOL balances (between) accuracy and computational cost for real-time monitoring. The study also provides insights into the suitability of different SSL model types for cloud anomaly detection, and allows studying the tradeoffs between these three aspects. Future research should focus on making the model more interpretable, federated learning for privacy preserving anomaly detection, as well as edge computing for real-time detection in the different clouds.