In this study, we developed a desktop agent system that automatically operates a GUI as desired by the user through natural language interaction between the user and the agent. In this system, the user can interact with the agent using speech recognition. The agent interprets the user’s input using GPT-3.5 and executes tasks. The agent interprets the user’s input using GPT-3.5 and using image analysis of the desktop screen, and executes tasks properly. Tasks are performed automatically and in the same way that users normally use GUI, e.g. by moving the mouse cursor, clicking icons, typing from the keyboard.