Many powerful anomaly detection algorithms are based on machine learning and rely on datasets for training and evaluation. However, anomalous samples are often rare in real-world datasets and might not be representative of anomalies encountered in the field. In this paper, we propose a synthetic anomaly generation methodology that focuses on generating large numbers of synthetic anomalies in images with defined variances in size, shape, and texture, achieving higher diversity scores than the state-of-the-art. To demonstrate the value of the proposed generation methodology for in-depth performance analysis, we generate anomalies in three MVTec AD datasets, which we then use to analyze and evaluate several anomaly detection algorithms. While all analyzed anomaly detection algorithms showed strong recall rates on these datasets, significant sensitivity differences regarding an anomaly’s size, shape, and texture are observable through the analysis with our synthetic datasets. While we observed some algorithms’ robustness towards different anomaly shapes and textures, others showed differences in recall rates of up to 80% points for some pixel manipulation methods. The results demonstrate the value of our synthetic anomalies, as they boost the capability to scrutinize anomaly detection algorithms.