Text segmentation is a crucial pre-processing step in the retrieval-augmented generation (RAG) process. By decomposing large-scale knowledge sources into manageable units, vectorizing them, and storing them in a vector data-base, retrieval accuracy can be improved through vector calculations while adapting to the context limitations of large language models (LLMs). Traditional chunking methods, which rely on fixed rules, lack semantic understanding, limiting the overall comprehension of long documents by LLMs and their ability to handle complex queries requiring information integration from multiple parts. RAPTOR addresses these challenges by constructing a multi-layer tree-shaped index through recursive clustering and summarization, effectively combining detailed and high-level information. However, RAPTOR's initial fixed-length chunking approach may scatter semantic focus within chunks, compromising embedding quality. This paper proposes an improvement: applying semantic segmentation technology to optimize the initial chunking stage. Our method dynamically identifies topic boundaries by calculating the semantic similarity of adjacent sentences, aggregating seman-tically coherent sentence groups into initial text chunks (leaf nodes). Experiments on the QuALITY dataset demonstrate that this approach enhances embedding and clustering effects, achieving a 5% increase in accuracy compared to the original method.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Enhancing RAPTOR in RAG Systems: Semantic Segmentation for Improved Initial Chunking

  • Xin Wan,
  • Xiaodong Xie,
  • Wei Lin,
  • Tingjun Xie,
  • Cheng Wang,
  • Yan liu,
  • Yi Pan

摘要

Text segmentation is a crucial pre-processing step in the retrieval-augmented generation (RAG) process. By decomposing large-scale knowledge sources into manageable units, vectorizing them, and storing them in a vector data-base, retrieval accuracy can be improved through vector calculations while adapting to the context limitations of large language models (LLMs). Traditional chunking methods, which rely on fixed rules, lack semantic understanding, limiting the overall comprehension of long documents by LLMs and their ability to handle complex queries requiring information integration from multiple parts. RAPTOR addresses these challenges by constructing a multi-layer tree-shaped index through recursive clustering and summarization, effectively combining detailed and high-level information. However, RAPTOR's initial fixed-length chunking approach may scatter semantic focus within chunks, compromising embedding quality. This paper proposes an improvement: applying semantic segmentation technology to optimize the initial chunking stage. Our method dynamically identifies topic boundaries by calculating the semantic similarity of adjacent sentences, aggregating seman-tically coherent sentence groups into initial text chunks (leaf nodes). Experiments on the QuALITY dataset demonstrate that this approach enhances embedding and clustering effects, achieving a 5% increase in accuracy compared to the original method.