Flaky tests (i.e. automated tests with a non-deterministic test outcome) undermine the trustworthiness of today’s DevOps build-pipelines, and recent research has investigated ways to detect or even remove flaky tests. In contrast, others proclaim that test engineers should “Assume all Tests Are Flaky” because, in today’s build-pipelines, one can never fully control all components of the system under test. Test engineers then capture the randomness of test results via what is called a flakiness score. In this paper, we extend an existing flakiness score to deal with system-level tests. We illustrate, via simulated test outcomes, how this refined score can support three different strategies for dealing with flaky tests—(i) Rerun, (ii) Fix and (iii) Monitor.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Extending a Flakiness Score for System-Level Tests

  • Joanna Kisaakye,
  • Mutlu Beyazıt,
  • Serge Demeyer

摘要

Flaky tests (i.e. automated tests with a non-deterministic test outcome) undermine the trustworthiness of today’s DevOps build-pipelines, and recent research has investigated ways to detect or even remove flaky tests. In contrast, others proclaim that test engineers should “Assume all Tests Are Flaky” because, in today’s build-pipelines, one can never fully control all components of the system under test. Test engineers then capture the randomness of test results via what is called a flakiness score. In this paper, we extend an existing flakiness score to deal with system-level tests. We illustrate, via simulated test outcomes, how this refined score can support three different strategies for dealing with flaky tests—(i) Rerun, (ii) Fix and (iii) Monitor.