Scholarlink

Introduction <p>Transformer-based large language models (LLMs) have transformed the field of natural language processing and led to significant advancements in various text processing tasks. However, the applicability of these LLMs in identifying related drug-adverse event (AE) pairs within clinical context may be limited by the prevalent use of non-standard sentence structures and grammar.</p> Method <p>Nine transformer-based LLMs pre-trained on biomedical domain corpora are fine-tuned on annotated data (<i>n</i> = 5088) to classify drug-AE pairs in unstructured discharge summaries as causally related or unrelated. These LLMs are then validated on text segments from deidentified hospital discharge summaries from Singapore (<i>n</i> = 1647). To assess generalisability, the models are validated on annotated segments (<i>n</i> = 4418) from the Medical Information Mart for Intensive Care (MIMIC-III) database. Performance of LLMs in identifying related drug-AE pairs is then compared against a prior benchmark set by traditional machine learning models on the same data.</p> Results <p>Using an LLM-Bidirectional long short-term memory (LLM-BiLSTM) architecture, transformer-based LLMs improve <i>F</i>1 score as compared to prior benchmark with BioM-ELECTRA-Large-BiLSTM showing an average <i>F</i>1 score improvement of 16.1% (increase from 0.64 to 0.74). Applying additional rules on the LLM-based predictions, like ignoring drug-AE pairs when the AE is a known indication of the drug, results in a further reduction in false positive rates with precision increases of up to 5.6% (0.04 increment).</p> Conclusion <p>Transformer-based LLMs outperform traditional machine learning methods in identifying causally related drug-AE pairs embedded within unstructured discharge summaries. Nonetheless the improvement in performance with rules indicates that LLMs still possess some degree of imperfection for this causal relation detection task.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Effectiveness of Transformer-Based Large Language Models in Identifying Adverse Drug Reaction Relations from Unstructured Discharge Summaries in Singapore

  • 期刊论文

  • 发表日期：

    2025年2月21日

查看全文
  • Yen Ling Koon,
  • Yan Tung Lam,
  • Hui Xing Tan,
  • Desmond Hwee Chun Teo,
  • Jing Wei Neo,
  • Aaron Jun Yi Yap,
  • Pei San Ang,
  • Celine Ping Wei Loke,
  • Mun Yee Tham,
  • Siew Har Tan,
  • Sally Leng Bee Soh,
  • Belinda Qin Pei Foo,
  • Zheng Jye Ling,
  • James Luen Wei Yip,
  • Sreemanee Raaj Dorajoo

摘要

Introduction

Transformer-based large language models (LLMs) have transformed the field of natural language processing and led to significant advancements in various text processing tasks. However, the applicability of these LLMs in identifying related drug-adverse event (AE) pairs within clinical context may be limited by the prevalent use of non-standard sentence structures and grammar.

Method

Nine transformer-based LLMs pre-trained on biomedical domain corpora are fine-tuned on annotated data (n = 5088) to classify drug-AE pairs in unstructured discharge summaries as causally related or unrelated. These LLMs are then validated on text segments from deidentified hospital discharge summaries from Singapore (n = 1647). To assess generalisability, the models are validated on annotated segments (n = 4418) from the Medical Information Mart for Intensive Care (MIMIC-III) database. Performance of LLMs in identifying related drug-AE pairs is then compared against a prior benchmark set by traditional machine learning models on the same data.

Results

Using an LLM-Bidirectional long short-term memory (LLM-BiLSTM) architecture, transformer-based LLMs improve F1 score as compared to prior benchmark with BioM-ELECTRA-Large-BiLSTM showing an average F1 score improvement of 16.1% (increase from 0.64 to 0.74). Applying additional rules on the LLM-based predictions, like ignoring drug-AE pairs when the AE is a known indication of the drug, results in a further reduction in false positive rates with precision increases of up to 5.6% (0.04 increment).

Conclusion

Transformer-based LLMs outperform traditional machine learning methods in identifying causally related drug-AE pairs embedded within unstructured discharge summaries. Nonetheless the improvement in performance with rules indicates that LLMs still possess some degree of imperfection for this causal relation detection task.