Text Segmentation
摘要
All components following the Preprocessing (Text Handling, Language and Topic Identification) are language specific, and need resources in the supported languages. The first group of analysis tools is intended to create word candidates (‘tokens’) from texts (paragraphs) as produced by the text handling: Sentence splitting, tokenisation, and normalisation.