The Statistical Engine of Experimentation

In Chapter 1 , we established the “why” of experimentation: the cultural shift from intuition-driven development to evidence-based decision-making. We defined the core concepts and vocabulary that allow us to frame business problems as testable hypotheses. Now, we move to the “how”—the rigorous mathematical machinery that powers the analysis of controlled experiments. This chapter is the statistical engine of the entire A/B testing framework. While Chapter 1 introduced the broader landscape of experimentation, including Multivariate Testing (MVT) and Multi-armed Bandits (MABs), this chapter will focus exclusively on the statistical foundation for the most common and fundamental method: the A/B test. The more complex statistical models for MVT, MABs, and the analysis of multiple treatment groups (e.g., using ANOVA) will be detailed in Part III, once this core foundation is firmly in place.