Enhancing Outlier Detection: A Hybrid Architecture with Autoencoder Clustering and Isolation Forest
摘要
Anomaly detection remains a prominent challenge in the field of data science, given its extensive range of applications and the array of methodologies at hand. Regrettably, numerous existing anomaly detection techniques exhibit shortcomings, such as ineffectiveness, non-intuitive behavior, or specialization for specific data types. In this research, we adopt a unique fusion of three distinct algorithms to effectively address the challenge of anomaly detection. Initially, we harness the power of autoencoders to reduce the dimensionality of the data, enhancing its manageability while preserving essential patterns. Subsequently, leveraging advanced clustering techniques, we partition our data into K distinct subsets, a strategic approach aimed at pinpointing local anomalies. Lastly, we employ the robust Isolation Forest algorithm to comprehensively identify remaining outliers in the dataset. This orchestrated fusion of methodologies promises to offer a comprehensive and efficient solution for detecting both local and global anomalies, Thus contributing to the development of anomaly detection methods in data science.