Aligning agent-based testing (ABT) with the experimental research paradigm: a literature review and best practices

The study of algorithmically curated media environments through emulated browsing has become a key method of computational social science. Here, we review underlying concepts and typical implementations of these so-called agent-based testing (ABT) studies, particularly those that experimentally vary inputs on search engines, social media, or aggregation platforms, to study subsequent outputs. This review aims to identify, evaluate, and discuss implementations and reporting standards of ABT studies within the experimental research paradigm. First, we discuss general assumptions of the experimental research paradigm and to what extent ABT experiments align with them, finding a considerable overlap with minor deviations. Second, we systematically reviewed 66 ABT studies from the social and information sciences, finding little reference to the experimental research paradigm vis-à-vis large variations in implementation and reporting practices. Third, our findings then inform five best-practice guidelines for future ABT experiments where we suggest to explicitly and much more strictly align those studies with the experimental research paradigm. We argue that ABT experiments would benefit from such an adaptation of already established experimental practices, thereby improving reproducibility and replicability.