Subword Segmentation

Starting from a word decomposition that uses morphemes, this chapter describes purely statistical methods. These methods use counts of pairs of characters and language models to derive limited vocabularies of subwords from large corpora. This chapter contains programs that implement the byte-pair encoding, WordPiece, and unigram algorithms.