Domain Adaptation for NER Using mBERT

This paper presents a comparative study of Kullback–Leibler (KL) and Jensen–Shannon (JS) divergence in enhancing NER using multilingual Bidirectional Encoder Representations from Transformers (mBERT) and Bidirectional Encoder Representations from Transformers (BERTs), by using domain-sensed embedding. By quantifying domain discrepancies, we assess how these divergences aid mBERT’s domain adaptation, aiming to improve its cross-lingual NER performance. The findings offer critical insights into optimizing domain-specific NER models, potentially transforming current adaptation methodologies. The study also underscores the significance of domain-specific word embeddings in capturing the unique semantics of different fields. Emphasizing the need for improvements, the paper suggests avenues like hybrid modeling, active learning, and advanced embedding techniques for contextual nuances. Incorporating the efficacy of KL divergence, our findings reveal that it outperforms JS divergence in refining mBERT for certain embeddings/methods, i.e., fast and simple domain adaptation for part-of-speech tagging “FLORS” NER tasks. The superior results with KL divergence for some of the embedding suggest that it more accurately captures the domain-specific nuances necessary for effective model adaptation. This pivotal insight could guide future research toward leveraging changes in the KL divergence so that it can be used as a standard in the domain adaptation toolkit for multilingual NER systems and adapted well to other embeddings.