Development of Novel Framework for Identifying Anomalies in High Volume of Data Using Robust Machine Learning Algorithm

Anomaly detection is a process that detects unlike observations from entire data points. Also, detection of anomalies in unlabelled data particularly is very tedious task using unsupervised learning models. It is a very big problem in banking, finance, telecom, cybersecurity where data are extremely high volume. Study of the model validation process in high-volume datasets is a critical task and also getting difficulty to get large-scale labelled data and evaluate the model performance. In real-world problem, in general, for unlabelled dataset, evaluation process not only takes time but also increase the project cost. To minimize this issue, here we proposed a novel framework merging with both the unsupervised and supervised learning process. In this framework, the process will generate global label followed by a majority voting ensemble approach. With the availability of global label, the framework latter established a classifier to train and finally predict the appropriate normal or anomaly label of any unlabelled data. This framework has seven stage or seven process and takes very low computing cost. To design this framework, we used five unsupervised learning models including Isolation forest, local outlier factor, Gaussian mixture, one-class SVM and auto-encoders; in addition to this, we used six supervised learning models like random forest, logistic regression, CART, Gaussian NB, K-nearest neighbour and XGBoost. All the applied model performance discussed extensively in the report. It is found the proposed framework has able to enhance the accuracy of anomaly detection rate.