State of the CArt: evaluating covering array generators at scale

Providing a reusable methodology for the evaluation of covering array generation utilities and apply it to a corpus of such tools, obtaining an overview of supported features and constraints, performance, output size, file formats, and practical considerations that may ease or hinder adoption.

Analysis of supported capabilities, input and output formats, and constraints, followed by an experimental evaluation of eight covering array generation tools, two of which were provided in updated versions, against a corpus of 295 publicly available benchmark models in six categories, producing arrays of strength two to six.

Capabilities, particularly constraint support, vary widely amongst competitors. Metaheuristic algorithms, commonly focused on postoptimization, tend to produce small arrays at the cost of performance. Approaches based on the In-Parameter-Order paradigm offer a good balance between speed and output size that may prove conducive to real-world adoption.

The choice of a covering array generation utility should be based on specific requirements related to the use case. Nevertheless, our evaluation identifies some candidates – CAgen, ACTS, and APPTS – which lead the field in terms of overall score. Others, such as PICT, offer unique features; however, a lack of standardization may lead to vendor lock-in.