Anomaly detection in large-scale cloud infrastructure is a necessary task that keeps system performance and security, and reliability in place. Rule based and supervised learning are unable to cope with the dynamic nature and large-scale challenges of the cloud environments because of the characteristics of the system. The purpose of this study is to present a self-supervised learning (SSL) framework of anomaly detection in cloud infrastructure by utilizing three state-of-the-art models: Bootstrap Your Own Latent (BYOL), Denoising Autoencoder (DAE), and Time series Generative Adversarial Network (TimeGAN). These are cloud performance metrics models (CPU utilization, memory usage, network traffic, disk I/O) that are used to analyze if there are anomalies in real time. To address this problem, the use of Amazon EC2 instances for data collection with data is achieved by a real-time streaming pipeline that uses Apache Kafka, Spark Streaming, and OpenTSDB. Experimental evaluations show that TimeGAN surpasses (93.5%) in detection accuracy, and BYOL balances (between) accuracy and computational cost for real-time monitoring. The study also provides insights into the suitability of different SSL model types for cloud anomaly detection, and allows studying the tradeoffs between these three aspects. Future research should focus on making the model more interpretable, federated learning for privacy preserving anomaly detection, as well as edge computing for real-time detection in the different clouds.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Self-supervised Learning for Anomaly Detection in Large-Scale Cloud Infrastructure

  • Maninder Pal Singh,
  • Vishnu Ravi,
  • Vineet Kumar Srivastava,
  • Nuzhat Noor Islam Prova

摘要

Anomaly detection in large-scale cloud infrastructure is a necessary task that keeps system performance and security, and reliability in place. Rule based and supervised learning are unable to cope with the dynamic nature and large-scale challenges of the cloud environments because of the characteristics of the system. The purpose of this study is to present a self-supervised learning (SSL) framework of anomaly detection in cloud infrastructure by utilizing three state-of-the-art models: Bootstrap Your Own Latent (BYOL), Denoising Autoencoder (DAE), and Time series Generative Adversarial Network (TimeGAN). These are cloud performance metrics models (CPU utilization, memory usage, network traffic, disk I/O) that are used to analyze if there are anomalies in real time. To address this problem, the use of Amazon EC2 instances for data collection with data is achieved by a real-time streaming pipeline that uses Apache Kafka, Spark Streaming, and OpenTSDB. Experimental evaluations show that TimeGAN surpasses (93.5%) in detection accuracy, and BYOL balances (between) accuracy and computational cost for real-time monitoring. The study also provides insights into the suitability of different SSL model types for cloud anomaly detection, and allows studying the tradeoffs between these three aspects. Future research should focus on making the model more interpretable, federated learning for privacy preserving anomaly detection, as well as edge computing for real-time detection in the different clouds.