Topics in Information Theory and Machine Learning

Linguistic symbols such as characters or words correspond to information. This chapter introduces the fundamental concepts of entropy, perplexity, and cross entropy to measure and compare the quantity of information carried by sequences of symbols. Using these concepts, a small dataset, and the scikit-learn machine-learning toolkit, we will create our first classifier based on decision trees.