Leaves are amongst the most sensitive parts of a plant. Any anomaly in the plant often affects the leaves first. Understanding the exact cause of leaf anomaly is a multi-faced challenge. We have come up with a novel end-to-end leaf anomaly detection system that differentiates between stressed and diseased leaf in the first stage using custom InceptionV3 model. Then it detects the exact type of disease and localizes the affected leaf area in the second stage using two YOLOv8 models. At last, it estimates the severity of the detected disease in the third stage using YOLO detections and IoU calculations. We have used Transfer Learning to use pre-trained weights. Our system is able to detect multiple anomalies on a single image. Classification task has 92% accuracy while Object Detection task and Severity estimation has about 80% accuracy.