Anomaly Detection (AD) is the task of identifying samples that do not conform to a learned model of normality. Prior work in deep AD is predominantly based on a familiarity hypothesis, where familiar features serve as reference in a pre-trained embedding space. While this strategy has proven highly successful, it turns out that it results in consistent false negatives when anomalies are caused by truly novel features that are not well captured by the pre-trained encoding – similar to a “blind spot”. We propose a new approach to AD using explainability to capture such novel features as unexplained observations in the input space. Hereby, we achieve state-of-the-art performance across various AD benchmarks by combining familiarity and novelty in a hybrid approach. Our approach has additional benefits such as eliminating the need for an outlier models. Inline with our motivation, we show that taking novel features into account reduces false negative anomalies by up to 40% on challenging benchmarks compared to methods relying solely on familiarity and provide visually inspectable explanations for pixel-level anomalies.