Understanding Spoken Language

This chapter describes models for speech recognition, i.e. for transferring spoken language into text. Speech recognizers use derived sound features for small time intervals as input. For speech processing, deep sequence-to-sequence models based on LSTM or transformers are used, which generate the recognized text. Alternatively, Convolutional Neural Networks are employed. A hybrid model of Sequence-to-Sequence and CNN models is able to achieve a lower recognition error than humans. When generating speech from text, WaveNet’s dilated CNN layers can reproduce the acoustic speech of a speaker extremely faithfully. Voice assistants, such as Siri and Alexa, allow users to engage in a dialogue. An example system is used to illustrate the construction of a variant of the Alexa voice assistant from subnetworks and other components. The classification of the events in a video is possible with variants of spatio-temporal convolutional layers. More difficult is the description of videos by subtitles, which can be done for example with the help of transformer translation models. In a last section, the influence of noise on speech recognition and the potential danger of adversarial attacks are discussed.