Obstetrics and gynecology (OB/GYN), branches of medicine that focus on pregnancy and the female reproductive system, heavily rely on ultrasound scanning. The automatic analysis of these images is an interesting tool as it can guide the sonographer in his diagnosis or provide similar images to the sonographer in real time. These tasks have become crucial because of the limited number of experts in the field, but deep learning methods in general have struggled to deal with them because of the lack of large annotated datasets for training. However, leveraging hierarchical rich annotations can be a way to alleviate this problem for learning better structured embedding spaces. In this vein, we propose a Semantic Abstraction Loss (SAL), which guides meta-embeddings to encode the information from the higher-order annotations in a Deep Metric Learning (DML) framework. We then build on the Expert Language Guidance (ELG) introduced by Roth et al. [21], that makes use of natural language captions to guide the visual similarities. We therefore propose an Ontology Language Guidance (OLG) that applies this concept to higher-level semantic annotations. Experimentally, we evaluate the impact of the integration of rich annotations through auxiliary embeddings or natural language on two visual similarity datasets: birds classification with CUB-200 and scan plane recognition on SUOG OB/GYN dataset.