This research analyzed the qualitative characteristics of messages in the Telegram messenger, which are used as input data for further analysis of text content. The parameters of these messages, such as their format, size, presence of noise, velocity, etc., have been carefully considered. The main goal of the article is to identify the optimal approach to saving a large amount of data before the important stage of text analysis. In the course of the study, a detailed analysis of literature sources devoted to this topic was carried out. The article examines the main advantages and disadvantages of existing data preprocessing algorithms, as well as problems related to data purity and their impact on potential research results. As part of software experiments, the influence of data preprocessing on the saved data size for further use, as well as the input data velocity, was evaluated. Among the proposed methods, the method of saving cleared tokens in string format and saving word codes in string format together with the word-code dictionary was highlighted. This is aimed at ensuring the effective distribution of tasks of the text analysis system during the day.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Optimizing Text Data Storage for Improved Natural Language Processing: A Comprehensive Analysis

  • Artem Kalancha,
  • Dmytro Uhryn

摘要

This research analyzed the qualitative characteristics of messages in the Telegram messenger, which are used as input data for further analysis of text content. The parameters of these messages, such as their format, size, presence of noise, velocity, etc., have been carefully considered. The main goal of the article is to identify the optimal approach to saving a large amount of data before the important stage of text analysis. In the course of the study, a detailed analysis of literature sources devoted to this topic was carried out. The article examines the main advantages and disadvantages of existing data preprocessing algorithms, as well as problems related to data purity and their impact on potential research results. As part of software experiments, the influence of data preprocessing on the saved data size for further use, as well as the input data velocity, was evaluated. Among the proposed methods, the method of saving cleared tokens in string format and saving word codes in string format together with the word-code dictionary was highlighted. This is aimed at ensuring the effective distribution of tasks of the text analysis system during the day.