Test automation is essential to ensure software reliability and speed up delivery cycles. However, flaky tests pose a significant challenge by giving inconsistent results even under identical conditions. Although previous studies have focused primarily on studying flakiness within unit tests and using open-source datasets, there is a gap in addressing flakiness in integration and system testing and within commercial software environments. This research aims to fill this gap. This paper presents a study on detecting flaky tests using machine learning (ML) techniques, leveraging a dataset compiled from six commercial projects with over 2500 Java-based test cases. We evaluated several ML models, including Decision Tree, Random Forest, Support Vector Machine and Neural Networks, and demonstrated that the Random Forest and Neural Networks models achieved the highest precision, recall, F1 scores and AUC-ROC values. These results highlight the effectiveness of ML models in improving flaky test detection and underscore their importance in commercial software environments. The proposed framework offers valuable information for improving the reliability of software testing, providing practical knowledge for researchers and practitioners in the field of software engineering.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

A Machine Learning-Based Detection of Test Automation Flakiness

  • Mohamed ElGazzar,
  • Eman Hossny,
  • Fatma A. Omara

摘要

Test automation is essential to ensure software reliability and speed up delivery cycles. However, flaky tests pose a significant challenge by giving inconsistent results even under identical conditions. Although previous studies have focused primarily on studying flakiness within unit tests and using open-source datasets, there is a gap in addressing flakiness in integration and system testing and within commercial software environments. This research aims to fill this gap. This paper presents a study on detecting flaky tests using machine learning (ML) techniques, leveraging a dataset compiled from six commercial projects with over 2500 Java-based test cases. We evaluated several ML models, including Decision Tree, Random Forest, Support Vector Machine and Neural Networks, and demonstrated that the Random Forest and Neural Networks models achieved the highest precision, recall, F1 scores and AUC-ROC values. These results highlight the effectiveness of ML models in improving flaky test detection and underscore their importance in commercial software environments. The proposed framework offers valuable information for improving the reliability of software testing, providing practical knowledge for researchers and practitioners in the field of software engineering.