Deep Learning-Based Preprocessing Tools for Turkish Natural Language Processing

As the demand for effective natural language processing applications in Turkish continues to rise, the need for text preprocessing tools tailored to the Turkish language increases. These tools form the initial step of any natural language application and improves the efficiency of complex tasks such as text summarization, question-answering, and machine translation. We propose a novel deep learning-based framework focusing on Turkish preprocessing tasks, including tokenization, sentence splitting, deasciification, part-of-speech tagging, vowelization, spell correction, and morphological analysis. The proposed framework is suitable for independent use of each preprocessing tool as well as the use in an all-in-one scheme. We use the CANINE model to train the character-level tools, and BERT and mT5 models for the token-based tools. We evaluate the framework for each task on the BOUN Treebank in the UD project and make both the tools and the codes publicly available.