Enhancing Data for Hard Anomaly Detection

Anomaly detection refers to the process of identifying elements, events, or observations in a given dataset that deviate from the expected patterns or exhibit distinct characteristics compared to other elements. We focus on a special and representative anomaly detection problem, called hard anomaly detection. The so-called hard anomaly has two characteristics in terms of the differences between abnormal and normal samples: (1) the boundary is exceedingly unobvious, and (2) the distribution is extremely imbalanced, both resulting in the difficulty of anomaly detection. It is really hard for regular classifiers to find the boundary between abnormal and normal samples here. To address the issue, we first propose a quantitative definition of the hard anomaly according to detection difficulty and then design a dedicated solution framework, named hard anomaly detection (HAD). Under HAD, we devise a GAN-based (generative adversarial network) method, called HadGAN, which produces both abnormal and normal samples with a similar distribution. We then pre-train a base anomaly detection model using the data generated by HadGAN. Moreover, we apply transfer learning to fine-tune the base model on real datasets. The superiority of our solution is demonstrated both by theoretical analysis and in extensive experiments.