Towards Mastering Real-World Robot Benchmarking: Lessons Learned from the Robothon Grand Challenge

In this paper, we delve into the pioneering insights gathered from the Robothon Grand Challenge (RGC) events of 2021–2023, focusing on the benchmarking of robotic manipulation skills. Central to our study is the innovative use of an internet-connected electronic task board (TB), a tool that allowed for the remote and comprehensive assessment of robot manipulation capabilities across varied research laboratories globally. We critically evaluate the pros and cons of this method, particularly in fostering reproducibility, enhancing transparency, and building trust in the results. Our findings not only contribute significantly to the field of robotics benchmarking but also pave the way for future advancements in the assessment and development of robotic systems under diverse conditions. This research underscores the evolving landscape of robotics, highlighting the role of smart technologies in shaping the future of robotics benchmarking and development.