Image annotations are indispensable to the development of new image-based AI software applications in medicine. The quality of these annotations, which serve as training data for AI methods, significantly impacts the effectiveness of AI-based methods. As a paradigm of the annotation process, we present wound annotation in epidermolysis bullosa (EB). EB is a group of rare genetic skin conditions that result in fragile skin and chronic wounds. Accurate wound segmentation is crucial for monitoring and treatment planning. The wound image annotations are made to train deep learning models for the automated segmentation of skin wounds caused by EB. The initial image annotations were widely different between experts with an average inter-annotator agreement of only 32%, as measured using the Dice similarity coefficient. By countermeasures such as annotation workshops to unify the annotation strategy, more precise annotation guidelines, and revised wound class definitions, we were able to distinctly increase the inter-annotator agreement to 59.3%. By sharing the encountered issues in the annotation process, we aim to help others anticipate and mitigate similar risks in medical image annotation tasks.