This paper presents Plant-AT, a novel hybrid attention model built for real-time plant disease categorization, which addresses a critical agricultural concern. Crop diseases in India cause yearly losses of 10–30% in tomatoes and 15–40% in grapes, putting food security at risk. Plant-AT combines Inverted Residual Blocks, an innovative LG-Attention Transformer, and a Robust Multi-Scale Fusion Mechanism, with key contributions such as Separable Self-Attention (SSA) for better global context understanding and Neighborhood Attention (NA) for precise local feature extraction. Plant-AT, with controlled 8.8 million trainable parameters, strikes a compromise between accuracy and efficiency, making it appropriate for deployment on edge devices with limited resources. This paper also introduces hybrid loss function which addresses class imbalance issues which are common in plant disease dataset.