Online controlled experimentation, and more specifically A/B testing, is an effective method for assessing the impact of software changes. However, when adopting A/B testing, a development team faces various organizational and technical challenges. In this paper, we propose a new notion of reusable controlled experimes (RCE) to simplify and accelerate the adoption of A/B testing for software teams. In its essence, an RCE is a reusable software component supplied with the built-in A/B testing functionality. We provide a proof-of-concept implementation of an RCE, integrate it into a mobile applications in the field of educational technology, and run an experiment to validate the proposed solution. We conclude by checking the resulting integration against the six criteria categories of Experimentation Evolution Model (EEM) to identify the maturity phase for each category. The resulting RCE is found to correspond to the Experimentation Evolution Model’s Walk maturity phase in three out of six categories, and to the Crawl phase in the other three categories.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

A/B Testing for Beginners: Reusable Controlled Experiments

  • Nursultan Askarbekuly,
  • Jean-Michel Bruel,
  • Frédéric Mayart,
  • Kamil Khairullin,
  • Bekzhan Talgat,
  • Yerzhan Gapurinov,
  • Gulnara Tulesheva,
  • Manuel Mazzara

摘要

Online controlled experimentation, and more specifically A/B testing, is an effective method for assessing the impact of software changes. However, when adopting A/B testing, a development team faces various organizational and technical challenges. In this paper, we propose a new notion of reusable controlled experimes (RCE) to simplify and accelerate the adoption of A/B testing for software teams. In its essence, an RCE is a reusable software component supplied with the built-in A/B testing functionality. We provide a proof-of-concept implementation of an RCE, integrate it into a mobile applications in the field of educational technology, and run an experiment to validate the proposed solution. We conclude by checking the resulting integration against the six criteria categories of Experimentation Evolution Model (EEM) to identify the maturity phase for each category. The resulting RCE is found to correspond to the Experimentation Evolution Model’s Walk maturity phase in three out of six categories, and to the Crawl phase in the other three categories.