Autonomous Human Computer Interaction System in Windows Environment Using YOLO and LLM

Imagine interacting with a computer as naturally as having a conversation with a friend. This paper introduces a Zero-Shot AI-powered approach in Human-Computer Interaction (HCI) within Windows environments, addressing current limitations in interface adaptability and user command execution. We utilize Tesseract Optical Character Recognition (OCR) for text extraction, Canny edge and contour detection for UI analysis, and the You Only Look Once (YOLO) model to recognize diverse UI elements. A key advancement is the integration of a Transformer-based Language Model (LLM), which plays a pivotal role in decision-making processes. The LLM interprets structured data, including coordinates of UI elements and text, to execute user commands within the Windows environment. This capability signifies a step towards systems that can be one of the foundational skills for future Artificial General Intelligence (AGIs). By translating images into structured data for LLM analysis, this system sets a precedent for AGIs to dynamically handle complex computer tasks, paving the way for more advanced, intuitive, and efficient computing domains.