Identifying Personal Identifiable Information (PII) in Unstructured Text: A Comparative Study on Transformers

Unstructured data encompasses a plethora of Personal Identifiable information (PII) represented in both standard (e.g. an email address) and unconventional forms (e.g. slang or emojis). Common examples, arise in social media posts but also more recently, in mesh data scenarios where multiple distributed data owners, might share fragments of their data to enable operations such as service compositions. Correctly identifying PII occurrences in such shared data, is crucial in ensuring that the participating entities adhere to the legal and ethical requirements of privacy legislation. In this paper, we present a toolbox of transformer models and comparatively assess the effectiveness and efficiency of such models in discovering PII in unstructured data. To this end, we evaluated state-of-the-art transformer models such as BERT, RoBERTa and XLNet using datasets containing upto 4.19 million tokens. Our results indicate that XLNet generates a lower false negative rate (0.059) than BERT (0.074) and RoBERTa (0.067). Comparative fine-tuning and evaluation among nine transformer models shows that the BERT models achieve 0.92 macro F1-score, a 5% performance improvement over previous works.