Camera-based systems in everyday applications require careful handling of privacy-sensitive image data. A promising approach to prevent data misuse is anonymization right after perception but before processing. However, traditional anonymization methods such as blurring and pixelation remove information essential for subsequent processing algorithms. Realistic anonymization, in contrast, can preserve vital information by generation of naturalistic replacements. Nevertheless, these systems pose unresolved questions regarding information preservation and correctness. We develop a systematic approach to analyze the effects of anonymization methods on model training and downstream tasks. Through a quantitative analysis of various anonymization methods, the challenges and pitfalls introduced by anonymization are highlighted. The lack of datasets for anonymization research, forces the adaption of inapt data. We use the state-of-the-art toolbox DeepPrivacy2 to generate realistic full-body anonymized data from COCO data and evaluate object classification using YOLOv10. Additionally, we demonstrate that appropriate evaluation of anonymization techniques requires specialized datasets. To address this gap, we introduce a handcrafted dataset, enabling us to prove the need for dedicated anonymization datasets. Based on systematically selected metrics, we assess the impact of anonymization on images or classes and present a range of experiments focusing on factors as object size and co-occurrence frequency with the anonymized class. Furthermore, novel findings on robustness of different model sizes and processing of anonymized images are presented. Our findings guide future directions in model adaptation to anonymized data, highlight improvements necessary in realistic anonymization generation, and underscore the importance of dedicated anonymization datasets.