We present a mathematical framework for discussing the class of divergence measures, which are essential tools for quantifying the difference between two probability distributions. These measures find applications in various fields such as statistics, machine learning, and data science. We begin by discussing the well-known Kullback–Leibler (KL) divergence, highlighting its advantages and limitations. To address the shortcomings of KL-divergence, the paper introduces three alternative types: \(\alpha \) -, \(\beta \) -, and \(\gamma \) -divergences. We emphasize the importance of choosing the right “reference measure,” especially for \(\beta \) - and \(\gamma \) -divergences, as it significantly impacts the results.