This study aims to develop a multimodal dialogue robot capable of having natural conversations with multiple users. To achieve this, we constructed a system that integrates the MMDAgent speech dialogue platform with a Kinect sensor. The proposed dialogue robot utilizes speech recognition and image processing to detect the direction of the sound source and the face orientation of each speaker, enabling it to identify the current speaker and determine the intended addressee. This allows the robot to direct its responses to the appropriate participant in real time. We implemented a communication mechanism that transmits the estimated face orientations and sound source directions to MMDAgent, which manages the dialogue logic. The system was evaluated through functional tests involving two users and three dialogue scenarios. In each case, the robot consistently generated appropriate responses based on the user’s position and gaze, confirming its effectiveness in handling dynamic multi-user interactions. This research contributes to the development of physically embodied robots capable of context-aware and socially intelligent behavior. In future work, we plan to extend the system to handle more than two users and integrate gesture recognition. We also aim to explore alternative image processing approaches such as OpenPose and OpenFace for more natural and robust human-robot interaction.