In this chapter, we work on a project aimed at building a storytelling language model from scratch in PyTorch. We start by explaining the foundations of language models, describing some architectures historically used in this regard. We then deep dive into the mathematics of the transformer architecture, which we will use later in our project.