Automatic number plate recognition (ANPR) systems play a vital role in global security and crime prevention. However, despite their importance, there is a notable lack of research assessing commercial off-the-shelf ANPR systems. Given that ANPR systems are not universally applicable due to regional variations in number plates, performance varies per region. A recent two-year study of a commercial ANPR system revealed a concerning average accuracy of 70%. This falls significantly short of the global average of 90%, compromising the reliability of the ANPR results. This study aims to address this knowledge gap by investigating factors that influence the performance of ANPR systems and comparing the effectiveness of commercial off-the-shelf solutions on South African vehicle plates. Four cloud-based ANPR solutions and an ANPR camera were assessed with a custom dataset. The custom dataset contains vehicle plates from different provinces and a government department. The study revealed that the camera viewing angle significantly affected the accuracy of the deployed ANPR system. Cloud-based ANPR APIs outperformed the ANPR camera system by 6.67%. In addition, most ANPR systems have difficulty detecting and correctly recognising number plate characters when there are multiple vehicles in the image. These findings underscore the need for region-specific optimisation and more comprehensive evaluation frameworks to enhance the reliability and accuracy of commercial ANPR systems in diverse real-world scenarios.