Considerable progress has been made recently in the development of methodologies, software libraries, problem sets and landscape analysis towards improving the benchmarking of black box optimisation algorithms. The majority of this work focuses on algorithm evaluation at a large/broad scale, by evaluating algorithms on large sets of different problem instances, across multiple dimensionalities. Broad evaluations are useful in providing a more complete picture of the behaviour of an algorithm. However they necessarily involve the summarization of experimental results, which can mean insights at the individual problem level are lost. This paper presents a case study of exploratory, per-instance algorithm evaluation, using real-world-representative problem instances from training small neural networks. The results illustrate that insights can be gained from looking at individual problem instances that would not be seen in a broad summative analysis. The work also highlights some potential problems that can exist in summative comparisons if care is not taken at the instance level.