Sequential Text Compression Strategy: A Study of NLP and Data Cleaning Preprocessing Tasks for Optimizing Token Utilization in LLMs
摘要
This paper highlights the relevance of applying Natural Language Processing (NLP) and Data Cleaning techniques to input prompts and their direct impact on reducing costs associated with the use of Large Language Models (LLMs). It analyzes how implementing fundamental preprocessing tasks in input prompts can reduce costs by up to 65% concerning the handling of tokens processed from user prompts, achieving up to 100% reliability from the LLM in some cases. The research was tested on gpt-4o-mini, gpt-4, and gpt-3.5-turbo models. Additionally, key performance indicators (KPIs) are included, along with their interpretation. This paper does not propose new preprocessing or text tokenization techniques; instead, it introduces a strategy called “Sequential Text Compression Strategy” to maintain the quality of outputs while optimizing resources. Finally, the paper discusses a new term called “Text-to-Token Compression Paradox”, which refers to the lack of a coherent relationship between the number of characters in a text and the number of generated tokens.