Aloe Vera is one of the therapeutic plant species grown in agriculture and it suffers from many kinds of leaf diseases, leading to decreased productivity. Accurate and timely disease identification is crucial for yield and quality enhancement. Traditional methods of spotting diseases are slow and require expert intervention. The Deep learning-based automated system increases accuracy & automation, and the system can be deployed on Edge AI devices and can be used for real-time monitoring. This paper focuses on recognizing herb leaf images which were collected from the Aloe vera plants and preprocessed using image enhancement methods. The diseases were classified as Xception models, which were optimized and deployed in an Edge AI device for real-time detection and prediction. It achieved enhanced precision in disease categorization versus the conventional approach. On the other hand, downsizing the model for on-edge-on-cloud deployment on Edge AI ensured lower latency with effective on-device inference. Data containing leaf images of Aloe Vera plants was processed and classified using the Xception model with accuracies reaching 94.41% training, 97.16% validation and 96.88% test accuracy. When comparing edge devices, NVIDIA Jetson Nano (922 ms latency) was found to be the best performing for deep learning inference, with the use of Raspberry Pi 4 (963 ms latency) also as an affordable option. The excessive latency of the Cortex M4F (80 MHz) made it inapplicable to real-time applications. The Edge AI system can detect diseases, with low latency and in real-time, allowing Aloe Vera farmers to be able to scale and use the best solution for the job.