Penalty-Aware Memory Loss for Deep Metric Learning

This paper focus on the deep metric learning, which has been widely used in various multimedia tasks. One popular solution is to learn a suitable distance metric by using triplet loss terms with sophisticated sampling strategies, However, existing schemes usually do not well consider global data distributions, which makes the learned metric suboptimal. In this paper, we address this problem by proposing a Penalty-Aware Memory Loss (PAML) that fully utilizes the expressive power of the combination of both global data distribution and local data distribution to learn a high-quality metric. In particular, we first introduce a memory bank to build the category prototype, that can capture the global geometric structure of data from the training data. The memory bank allows imposing a penalty regularizer during the training procedure without significantly increasing computational complexity. Subsequently, a new triplet loss with softmax is defined to learn a new metric space via the classical SGD optimizer. Experiments on four widely used benchmarks demonstrate that the proposed PAML outperforms state-of-the-art methods, which is effective and efficient to improve image-level deep metric learning.