What Data Scientists (Care To) Recall

To maintain and evolve a software system, developers need to gain new or recover lost knowledge about that system. Thus, program comprehension is a crucial activity in software development and maintenance processes. We know from previous work that developers prioritize what information they want to remember about a system based on the perceived importance of that information. However, AI-based software systems as a special case are not developed by software developers alone, but also by data scientists who deal with other concepts and have a different educational background than most developers. In this paper, we study what information data scientists (aim to) recall about their systems. For this purpose, we replicated our previous work by interviewing 11 data scientists, investigating the knowledge they consider important to remember, and whether they can remember parts of their systems correctly. Our results suggest that data scientists consider knowledge about the AI-project settings to be the most important to remember and that they perform best when remembering knowledge they consider important. Contrary to software developers, data scientists’ self-assessments increase when reflecting on their systems. Our findings indicate similarities and differences between developers and data scientists that are important for managing the processes surrounding a system.