The development of autonomous systems capable of executing complex tasks in computational environments represents a significant research direction in the field of artificial intelligence, more specifically in the subfield of machine learning. In the current context, intelligent agents that can learn and adapt control strategies to perform repetitive or complex tasks are becoming increasingly important. This applies to both Reinforcement Learning-based models and, more recently, to Large Language Models (e.g., GPT-3, PaLM). This paper explores the use of reinforcement learning algorithms, specifically the implementation of Proximal Policy Optimization (PPO), to train agents capable of performing tasks from the MiniWob++ suite. This paper moves away from the classical paradigm of web control and seeks to re-frame the problem in a way that could transfer better to real-world applications, by having the agent only observe the environment through screen pixels and interact with the world using low-level cursor control, akin to how humans use computers. In addition, this paper proposes architectural changes and training strategies to tackle this proposed environmental setting. Experimental results highlight that learning such control policies is feasible while using no human-annotated data and working with a harder problem formulation.