Detecting Asthma Presentations from Emergency Department Notes: An Active Learning Approach

Emergency department (ED) triage notes contain valuable information for near real-time syndromic surveillance of emergent public health events. With the increasing use of machine learning algorithms for classification of ED triage notes, active learning (AL) offers a way to automatically obtain high-quality data for labelling, resulting in a reduced requirement for annotators to search for suitable data. Active learning involves selecting the most valuable data to enhance the model’s learning process, guided by a query strategy that identifies samples with most impact on the model’s training. The objective of this study was to assess the effectiveness of active learning in developing a high-performing model for ED syndrome detection. The research aimed to explore pool-based active learning and investigated various query strategies to improve the model’s performance in identifying asthma presentations from ED triage notes. Our results showed that AL can be highly effective for reducing annotation effort while building reliable models. Uncertainty sampling strategy outperformed all other methods, achieving an F1 score of 0.91 to improve the baseline score by 7.6 percentage points.