A/B Testing for Beginners: Reusable Controlled Experiments
摘要
Online controlled experimentation, and more specifically A/B testing, is an effective method for assessing the impact of software changes. However, when adopting A/B testing, a development team faces various organizational and technical challenges. In this paper, we propose a new notion of reusable controlled experimes (RCE) to simplify and accelerate the adoption of A/B testing for software teams. In its essence, an RCE is a reusable software component supplied with the built-in A/B testing functionality. We provide a proof-of-concept implementation of an RCE, integrate it into a mobile applications in the field of educational technology, and run an experiment to validate the proposed solution. We conclude by checking the resulting integration against the six criteria categories of Experimentation Evolution Model (EEM) to identify the maturity phase for each category. The resulting RCE is found to correspond to the Experimentation Evolution Model’s Walk maturity phase in three out of six categories, and to the Crawl phase in the other three categories.