In many critical machine learning applications, the lack of transparency and interpretability of black box models raises concerns about their trustworthiness. This issue highlights the importance of understanding the causality behind predictions, which is related to endogeneity, causal inference, and transfer learning. Accurately estimating the true coefficients and determining the important feature set is necessary for accurate attribution. To address this challenge, we propose a new explainable model called Stable Local Interpretable Surrogate (SLIS), which learns the causal structure and provides trustworthy local interpretation. SLIS exploits the conditional expectation invariance structure and environmental heterogeneity to determine causal relationships. It can select truly important features without requiring additional domain knowledge. We provide the theoretical basis for this technique and confirm its effectiveness on synthetic and real-world datasets. Compared to existing methods, SLIS provides robust and stable estimates of the local surrogate model, a simple and interpretable model that approximates the behavior of a black-box model.