Sterilization centers play a fundamental role in hospitals worldwide and are responsible for disinfecting and washing contaminated equipment used in surgical and other medical procedures. The staff is in close contact with contaminated equipment and rely on robotic systems to perform tasks in the sterilization centers. In the AL10 system developed by KEN Hygiene Systems, the staff currently uses a tablet, which poses the risk of cross-contamination. This paper reports on our experience employing automatic speech recognition and large language models to enable users to effectively communicate with the AL10 system through voice commands instead. The developed speech-based interface is designed as an offline add-on module to the AL10 system and allows users to specify tasks with minimal language constraints. We evaluated the solution with end users in a real-world environment at a Danish hospital and the results highlighted that they were optimistic about the new system, indicating its potential to be deployed as a product in the sterilization centers operated by KEN. Overall, our solution provides a significant contribution by investigating the potential of offline speech-based interfaces in the context of healthcare robotic systems.