Automated Vickers hardness measurement methods must be robust and versatile to cope with materials with rough and noisy patterned surfaces. The behaviour of several state-of-the-art algorithms for automated Vickers hardness measurement is evaluated and compared for their respective robustness and accuracy. The evaluation is based on real world industrial data, collected in a Vickers indentation dataset. Addressing the current lack of publicly available (annotated) Vickers indentation data, and in order to provide a benchmark for further investigations and experiments, we release the databases used in this work, together with expert groundtruth covering diagonal measurement and segmentation masks, and make them available on-line for public open access.