The present era has witnessed a stark proliferation in the amount of text data and its analysis. With the present pace of digitalization, effective text processing is paramount for various natural language processing (NLP) applications. This research aims to introduce a versatile Python library that leverages fine-tuned large language models (LLMs) to perform a suite of text processing tasks, including grammar correction, profanity filtering, masking, tokenization, synonym replacement, stopword removal, and named entity recognition (NER). The intention was to explore the possibilities of automation of the manual pre-processing tasks using Self-supervised learning methods. The library encapsulates each function performing a specific task into a wrapper class, thus ensuring modularity and scalability. The models are built on top of state-of-the-art open-source architectures such as Google’s T5 architecture and DistilBERT. Additionally, the library provides a bifurcation to support both small and large models. This provides a balance to the accuracy and speed tradeoff by providing the ultimate model choice to the user. The library streamlines and automates the tedious and manual process of text data pre-processing, making it significantly more user-friendly and efficient.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

TextPrep: Automating the Process of Text Data Pre-processing by Leveraging Self-supervised Learning

  • Archit Anand,
  • Kumar Sampurn,
  • Tanya Bajaj,
  • A. Lakshanya,
  • R. Bharathi

摘要

The present era has witnessed a stark proliferation in the amount of text data and its analysis. With the present pace of digitalization, effective text processing is paramount for various natural language processing (NLP) applications. This research aims to introduce a versatile Python library that leverages fine-tuned large language models (LLMs) to perform a suite of text processing tasks, including grammar correction, profanity filtering, masking, tokenization, synonym replacement, stopword removal, and named entity recognition (NER). The intention was to explore the possibilities of automation of the manual pre-processing tasks using Self-supervised learning methods. The library encapsulates each function performing a specific task into a wrapper class, thus ensuring modularity and scalability. The models are built on top of state-of-the-art open-source architectures such as Google’s T5 architecture and DistilBERT. Additionally, the library provides a bifurcation to support both small and large models. This provides a balance to the accuracy and speed tradeoff by providing the ultimate model choice to the user. The library streamlines and automates the tedious and manual process of text data pre-processing, making it significantly more user-friendly and efficient.