Family environment is one of main area for service robot applications. Complex multi-modal scene and users’ high requirements for service levels are challenges that service robots face. In order to improve the service level of service robots in multi-modal family scene, this paper proposes a service robot task reasoning mechanism that integrates multi-modal information and ontology knowledge. The system manages user and environment information by ontology knowledge base, obtains fused multi-modal information including vision, voice, and scene knowledge in real time, and reasons service task based on fine-tuned LLM (Large Language Model). The system is designed based on the edge-cloud collaborative architecture and has high service levels in experimental tests of multi-modal family elderly care scene.