Understanding AI Better with Brain Research

The black-box problem, referring to the insufficient understanding and difficult interpretation of deep neural networks, remains a significant challenge for AI. The EU and DARPA emphasize the necessity for transparency and explainability of AI. Neuroscience methods could help to better understand and interpret artificial neural networks. In brain research, a lesion refers to damage to a part of the nervous system, providing valuable insights into the functions of various brain regions. A similar approach is used in AI research to understand the role of different components in artificial neural networks. This is achieved by removing or altering certain neurons, layers, or connections and observing the resulting changes in the network’s output or performance. Various network visualization techniques also provide insights into the functions of deep neural networks. Neurons can specialize in detecting specific features, while others respond to more abstract concepts. Feature visualization is another method based on neuroscience concepts, using an optimization process to maximize the activation of a particular neuron or layer. A related method, layer-wise relevance propagation, traces the model’s output back to the input layer, assigning relevance ratings to each neuron and feature of the input.