Although deep neural networks exhibiting superior performance across numerous tasks, their application in high-risk domains is limited by a lack of interpretability and trustworthiness. In this paper, an interaction value calculation method is firstly proposed, which faithfully represents the interaction utility of each variable in the feature map. Secondly, an interpretable method for top-down construction of interaction hierarchy graph based on interaction utility is proposed to understand the visualized knowledge represented by filters and to elucidate the decision-making process of the network. Extensive experiments were carried out on publicly available datasets and models that had been pre-trained. Experimental results indicate that each node in the graph consistently corresponds to the same part of an object across various images belonging to the same category. The faithfulness evaluation shows that the filters involved in the graph nodes have a major role in the network. Furthermore, the quantitative evaluation shows that our method improves over the others by an average of 0.18%, 1.19%, and 2.18% on the EBPG, mIoU, and Bbox metrics, respectively.