Case-based Reasoning (CBR) relies on the quality of similarity metrics. Here, current developments in eXplainable Artificial Intelligence (XAI) aid in providing insights about their behavior and performance. While explainability for AI models has gained importance, XAI for time-series data models remains less developed compared to those for more common data types. The diversity of XAI approaches makes it challenging to identify fair metrics that effectively assess the quality of explanations. Furthermore, XAI methods are often highly customizable, and small changes in their parameters can significantly impact the quality of the explanations. In this work, we propose an adaptation of existing explainability metrics to extend their application to time series data models that can be used for the retrieval of cases. To validate the significance of the metrics, we propose a user-driven study to examine the correlation between human preferences and metric quality.