Multi-armed Bandits: Balancing Exploration and Exploitation

In a classic A/B test, we commit to a period of pure exploration. We gather data on all variants, even those that are underperforming, to achieve statistical confidence. Only after the experiment concludes do we exploit our findings by rolling out the winning variant to 100% of users. This separation of exploration and exploitation is powerful for making confident, long-term decisions, but it comes at a cost: regret. Regret is the value lost by continuing to show users an inferior variant during the exploration phase.