<p>As software systems become increasingly complex, the application of artificial intelligence (AI) in software engineering is gaining relevance. However, a critical gap exists in the understanding and interpretation of AI-driven decision-making processes, especially in areas intrinsically linked to human expertise, such as software modeling. This paper proposes an exploratory study on the feasibility, efficacy, and relevance of eXplainable Artificial Intelligence (XAI) techniques within this context. The application of machine learning (ML) to software models is relatively recent, so efforts such as the ModelSet dataset are crucial for deriving effective training data for ML models. In addition, predictive software modeling tasks present some unusual requirements, such as the need for multi-class and multi-label approaches that are not as commonly investigated from an XAI perspective. In fact, the adoption of XAI in software modeling has been barely explored and possibly requires adapted methodologies. Our approach encompasses an in-depth examination of explanations generated by five XAI techniques that evaluate feature contributions globally for ML models and locally for specific predictions. This could help software modelers understand, for example, why a model is classified in a particular domain. Additionally, our study includes a survey conducted among software modelers to capture how explanations support their decision-making, to evaluate the perceived level of agreement between different XAI techniques, and to identify current limitations. We argue that XAI can improve the transparency and trustworthiness of the decision-making process for software modelers, thereby fostering a deeper understanding of intricate modeling tasks.</p>