The analysis of complex, weakly labeled data is increasingly popular, presenting unique challenges. Traditional unsupervised clustering aims to uncover interrelated sets of objects using feature-based similarity of the objects, but this approach often hits its limits for complex multimedia data. Thus, semi-supervised clustering that exploits small amounts of labeled training data has gained traction recently. In this paper, we propose LabeledPAM, a semi-supervised extension of FasterPAM, a state-of-the-art k-medoids clustering algorithm. Our approach is applicable in semi-supervised classification tasks, where labels are assigned to clusters with minimal labeled data, as well as in semi-supervised clustering scenarios, identifying new clusters with unknown labels. We evaluate our proposal against other semi-supervised clustering techniques suitable for arbitrary distances, demonstrating its efficacy and versatility.