AI-Assisted Hate Speech Moderation—How Information on AI-Based Classification Affects the Human Brain-In-The-Loop

Every day, social media content moderators must decide within seconds hundreds of times whether or not user-generated content constitutes hate speech. Although IS research is making continual progress in automatically detecting potential hate speech content through AI-assisted processing, the final decision still resides in the human-in-the-loop. To support the content moderators, the results of AI-based classifications are regularly displayed during the decision-making process—but is this advisable? To approach an answer, the neural and behavioral effects of two opposing AI-based classifications are tested against each other. The results from a fNIRS experiment show that opposing AI-based classifications leads to different cortical activation patterns, which in turn depend on the individual’s importance of hate speech prevention. Moreover, this exploratory study indicates that AI-based classifications may also induce a “cortical relief” seemingly cause behavioral effects that at least cast doubt on the validity and desirability of the AI-assisted human decision.