As agriculture faces growing challenges from climate change, pests, and diseases, effective monitoring systems are crucial for safeguarding crops and improving productivity. This paper introduces AgriGuardNet, an advanced hybrid framework designed to detect both plant diseases and pests simultaneously from a single input. By leveraging the Xception model for disease classification and a hybrid architecture combining EfficientNetB0 and MobileNetV2 for pest detection, AgriGuardNet efficiently performs multi-task learning while optimizing resource usage. The model achieved impressive results, with disease detection accuracy and pest detection accuracy on the validation set, culminating in a combined accuracy of 97.88%. Additionally, the model was quantized for deployment on edge devices, maintaining near-original performance while minimizing computational requirements. Furthermore, AgriGuardNet was integrated with real-time IoT systems for continuous crop health monitoring and pest management, enabling automatic detection and notifications to farmers. This real-time integration enhances decision-making by providing immediate feedback for early intervention. Through extensive training, fine-tuning, and performance evaluations, AgriGuardNet outperforms existing solutions, offering a scalable, adaptable, and practical tool for comprehensive agricultural monitoring in resource-constrained environments.