Breaking Content Moderation

Generative AI removes barriers to media production while drastically increasing its scale and speed, quickly outpacing existing moderation models. Unlike typical memes, generative models can produce endless, pixel-level permutations of prompts, evading detection by automated systems designed to flag identical content. This ability, along with new variants like subliminal hate, creates new avenues for harmful content to bypass moderation. This chapter interrogates the dilemma of human versus machinic content moderation. While human moderators are more effective at detecting coded, ambiguous, or subliminal forms of hate—the scale of the task poses not only logistical problems but also significant risk to mental well-being at a societal level. On already-struggling platforms, researchers are now exploring AI models for moderation, suggesting that the automation of hate may only be effectively addressed through automated mitigation.