Intrinsic Dimensionality

We have seen that high dimensionality poses difficulties for vector retrieval. Yet, judging by the progression from hand-crafted feature vectors to sophisticated embeddings of data, we detect a clear trend towards higher dimensional representations of data. How worried should we be about this ever increasing dimensionality? This chapter explores that question. Its key message is that, even though data points may appear to belong to a highdimensional space, they actually lie on or near a low-dimensional manifold and, as such, have a low intrinsic dimensionality. This chapter then formalizes the notion of intrinsic dimensionality and presents a mathematical framework that will be useful in analyses in future chapters.