Fine-tuned multimodal large language model for autonomous state cognition system of shape-recognition 6-bar tensegrity integrated with flexible sensors

Conducting dynamic exploration in complex and unpredictable environments, particularly in space exploration, reveals the great potential of systems based on tensegrity structures. However, the implementation of such systems faces a series of intelligent challenges, including the reliability of wireless monitoring, the efficiency of human-computer interaction, and the optimization of intelligent analysis and recommendation capabilities. In this study, we introduce a 6-bar tensegrity system equipped with 24 flexible sensors, leveraging fine-tuned multimodal large language model to enable autonomous state cognition system including self-shape recognition, alarm system, as well as fault diagnosis. Supported by long and short-term memory models, the tensegrity is able to reconstruct its own shape via conductive flexible tendons without relying on external sensors. By combining the flask server with the fine-tuned large language model, the tensegrity automatically transmits data to the iPhone for wireless monitoring. Finally, we developed the fine-tuned LLM and employed it to facilitate fault diagnosis and human interaction, enabling users to effectively obtain the requisite information through natural language processing techniques. This autonomous state cognition system relying on tensioning bars shows great potential for future exploration and becomes a powerful tool for multifunctional applications in the real world.