This paper presents an approach (called Hino) to detect outliers present in a data set, also called aberrations or anomalies. These data may reduce the quality of data analysis and lead to erroneous results. In the case of learning algorithms, they can deviate their behavior, i.e. reduce their efficiency. Thus, outlier detection is crucial where it improves performance by providing better data quality and reduces the influence of outliers. Inter Quartiles Range (IQR) is a popular statistical detection method, which has the advantage of being simple and fast in calculation time. It is based on the distribution quartiles of a data set and considers the most extreme values as outliers. This means that this method only searches for point outliers, which is a restrictive and naive approach. Indeed, nothing prevents an element from having an extreme value while remaining consistent with the rest of the elements. The proposed method is also a statistical detection method based on quantiles, but it looks for contextual outliers instead of point outliers and consider the context of a point to determine whether it is an outlier or not. The effectiveness of Hino is compared with the original IQR method and other approaches, including Isolation Forest, SVM, and LOF, using 16 real and 278 synthetic data sets.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Hunting Inside N-Quantiles of Outliers (Hino)

  • Jessy Colonval,
  • Fabrice Bouquet

摘要

This paper presents an approach (called Hino) to detect outliers present in a data set, also called aberrations or anomalies. These data may reduce the quality of data analysis and lead to erroneous results. In the case of learning algorithms, they can deviate their behavior, i.e. reduce their efficiency. Thus, outlier detection is crucial where it improves performance by providing better data quality and reduces the influence of outliers. Inter Quartiles Range (IQR) is a popular statistical detection method, which has the advantage of being simple and fast in calculation time. It is based on the distribution quartiles of a data set and considers the most extreme values as outliers. This means that this method only searches for point outliers, which is a restrictive and naive approach. Indeed, nothing prevents an element from having an extreme value while remaining consistent with the rest of the elements. The proposed method is also a statistical detection method based on quantiles, but it looks for contextual outliers instead of point outliers and consider the context of a point to determine whether it is an outlier or not. The effectiveness of Hino is compared with the original IQR method and other approaches, including Isolation Forest, SVM, and LOF, using 16 real and 278 synthetic data sets.