LLMs Red Teaming

Prompt red-teaming is a form of evaluation that involves testing machine learning models for vulnerabilities that could result in undesirable behaviors. It is similar to adversarial attacks, but red-teaming prompts appear like regular natural language prompts, and they reveal model limitations that can cause harmful user experiences or aid violence. Red-teaming can be resource-intensive due to the large search space required to search the prompt space of possible model failures. Augmenting the model with a classifier trained to predict potentially undesirable texts is a possible workaround. Red-teaming LLMs is a developing research area, and there is a need for best practices, including persuading people to harm themselves or others and other problematic behaviors, such as memorization, spam, weapons assembly instructions, and the generation of code with pre-defined vulnerabilities. The challenge with evaluating LLMs for malicious behaviors is that they are not explicitly trained to exhibit such behaviors. Therefore, it is critical to continually develop red-teaming methods that can adapt as models become more powerful. Multi-organization collaboration on datasets and best practices can enable smaller entities releasing models to still red-team their models before release, leading to a safer user experience across the board.