The Automatic Seaman: From Speech2text to Text2Task

This paper introduces the development of an autonomous seaman system, leveraging sound-to-text processing, intent detection, slot filling and action control. The Whisper model was employed to process sound commands and transcribe them, while JointBert was used to extract intentions and fill relevant slots. For enhanced model performance, Whisper model was fine-tuned using real voice data in the Portuguese language, while JointBert benefited from data generated by Chat GPT-3. To ensure effective interaction management and action execution, a directed graph structure was used as abstraction. The system’s performance was evaluated based on word error rate, intent accuracy, F1 score for slot filling and task accomplished. Experimental results showcased the effectiveness of our proposed approach, demonstrating accurate comprehension of sound commands and efficient action control. As a result, the autonomous seaman robot holds great promise for practical applications in automating diverse seafaring tasks. The improvement in the man-machine interface is very relevant for manned systems, but even more for unmanned robotic systems.