Interpretability in multi-output regression models is still an under-explored area in explainable AI. This work proposes an effective method for comprehending feature importance for specific targets in multi-output regression problems. By leveraging the power of Local Interpretable Model-Agnostic Explanations (LIME), we produce sample-specific distances for each target and incorporate them through dimensionality reduction methods like t-Distributed Stochastic Neighbor Embedding (t-SNE). To cluster the samples in the embedding space, the LIME explanation vectors are clustered individually for each target, and overlapping clusters are identified to recognize common influential aspects. Hence, instead of finding standard feature explanations, clustering based on the explanation vectors can determine the effects of characteristics across various targets, enabling us to gain better insights and make more targeted decisions. We measure cluster cohesion using silhouette scores and DB Index, and examine predominant feature patterns among overlapping groups, i.e., samples that are in the same clusters for both targets. This approach presents a novel and interpretability-focused form for comprehending challenging multi-target prediction models.