A Longitudinal Study on the Effect of Patches on Code Coverage and Software System Maintainability

Contemporary software development often involves the use of source control repositories which are hosted online and making incremental patches (commits) throughout the development process. Online hosting services facilitate the use of build pipelines and the integration of code coverage services into these pipelines. However, existing research into how incremental patches to software systems affect code coverage has not fully taken advantage of the data that is made available by these coverage services. This paper presents a partial replication of two previous studies on patch coverage, analysing over 50, 000 builds from 46 projects obtained from two popular coverage services, Codecov and Coveralls. Data quality issues, such as missing commits, duplicate builds from Cron Jobs, and sudden coverage drops, were identified and addressed, highlighting the need for rigorous data cleaning process when mining data from coverage services. Results indicate that patches are generally either fully covered or entirely uncovered, with a majority achieving full coverage, suggesting that very seldom do engineers opt for partial coverage. There is a weak correlation (correlation coefficient: 0.23) between patch coverage and system coverage, indicating that patch coverage alone cannot be used to predict system coverage evolution. Furthermore, patch testing does not enhance patch maintainability.