Disaster Tweet Classification Using Fine-Tuned Deep Learning Models Versus Zero and Few-Shot Large Language Models

The task of effectively filtering and categorizing the vast volume of user-generated content on social media during disaster events is crucial for emergency management and disaster response organizations to allocate their resources efficiently. Deep learning approaches, including recurrent neural networks, Capsule Neural Networks (CapsNets) and transformer-based models, have been successfully used for text classification. However, they have not been extensively compared in the context of crisis tweet classification. Furthermore, such models have not been compared with foundational large language models (LLMs), such as the open-source Llama 2, which are known to produce good results in zero and few-shot settings. In this study, we conducted a comprehensive comparison between CapsNet models, BERT models, and Bi-LSTM, one of the widely used recurrent neural network models, in the task of classifying crisis tweets. We also compared these models with Llama 2 in 0-shot, 1-shot, and 5-shot learning settings. We considered two classification tasks: binary classification for informativeness and multi-class classification for humanitarian content. For this purpose, we used several benchmark datasets for crisis tweet classification, namely CrisisBench, CrisisNLP, and CrisisLex. Our experiments revealed that the results of the CapsNet models are comparable to those of the Bi-LSTM models across various metrics. However, BERT models consistently outperformed both Bi-LSTM and CapsNet models across different datasets for both classification tasks. Notably, Llama 2 exhibited promising performance, considering its exposure to no data instances in the 0-shot and only few instances in 1-shot and 5-shot learning settings, showcasing its potential for handling crisis tweet classification. However, it is important to highlight that the fine-tuned models, including CapsNet, BERT, and Bi-LSTM, achieved superior performance overall, emphasizing the influence of fine-tuning on model effectiveness. These findings suggest that BERT is still one of the most effective models for classifying crisis tweets. Additionally, Llama 2, with its strong performance in limited-shot scenarios that match the scenarios in the crisis domain, presents an intriguing path for further exploration in handling crisis-related content.