A Gradient-Based Approach to Interpreting Visual Encoding Models

The field of Computational Neuroscience has boomed over the past few decades. Research has been focused on devising neuroscientific models with state-of-the-art performance in replicating neural responses. A significant role is played by Deep Neural Networks (DNNs) in this pursuit. However, DNNs have a drawback in terms of interpretability. DNNs are infamous for their ‘black box’ nature that diminishes their usefulness in neuroscientific interpretability. In this article, we have proposed a gradient-based approach to address this issue. The proposed Voxelwise Stimulus Optimization Algorithm optimizes the stimulus image by backpropagating the loss calculated from the neural responses. The optimization algorithm generates an activated image map that determines the areas responsible for the predicted neural response. In other words, it develops a more generalized stimulus-response relationship. The study uses an AlexNet model and generates an activation map for all stimulus images from the Kay 2008 dataset based on the responses from the Lateral Occipital Lobe. The result obtained suggests that the shape-based features are responsible for activating the Lateral Occipital Lobe. The observation is consistent with previous literature, acting as a validation for our proposed algorithm.