Agriculture is fundamental to ensuring global food security and economic stability but faces mounting challenges due to climate change, resource scarcity, and the increasing prevalence of pests and diseases. Effective monitoring and timely intervention are critical to addressing these issues and enhancing agricultural productivity. In this research, we present an innovative framework that leverages advanced deep learning architectures to detect plant diseases and pests from distinct input sources. Our approach leverages the Xception model for disease classification and a hybrid architecture combining EfficientNetB0 and MobileNetV2 for pest detection, enabling efficient and specialized task performance while optimizing resource utilization. The disease detection model attained an accuracy of 99.21%, while the pest detection model reached 98.40%. To optimize for edge device deployment, the models were quantized, resulting in only a very minimal drop in accuracy, ensuring that the models remain efficient while retaining near-original performance. Through comprehensive training, meticulous fine-tuning, and rigorous performance evaluations, we showcase substantial advancements in detection accuracy over existing models. With their scalable and adaptable design, these models offer robust solutions for diverse crops and environmental conditions, providing comprehensive tools for crop health monitoring and pest management.