With rising demand for energy efficiency and sustainability, advanced technologies to optimize energy consumption in various domains are in high demand. This paper proposes an Edge-based Smart Energy Management System that intelligently monitors and controls energy usage in real-time using object detection techniques. This system employs a distributed architecture to efficiently manage energy consumption by combining the power of edge computing and computer vision. Smart computer vision devices are deployed at the edge to collect real-time data from the environment. These devices use YOLOv8 object detection algorithms to identify and track energy-intensive objects such as appliances like lights, and fans. The detected objects information is then processed and analyzed at the edge devices, which generate the corresponding threshold value for managing the electrical appliances. These processed data will be sent to the cloud infrastructure for further analysis and decision-making. Through the integration of object detection and YOLOv8 machine learning techniques, this system offers several key benefits. Firstly, it enables real-time monitoring of energy consumption at a granular level, allowing users to identify the number of persons and take appropriate actions to reduce electricity consumption. Secondly, the system provides accurate energy forecasting, enabling proactive load management and efficient allocation of energy resources. The proposed system will implement the computer vision in real-time monitoring and evaluation to assess its performance, scalability, and energy-saving potential. The evaluation results will demonstrate the proposed system’s effectiveness in optimizing energy usage, reducing costs, and contributing to a more sustainable environment.