Sustainable smart agriculture forms one of the focal points in Society 5.0. We propose an AI-IOT enabled framework capable of processing multi-modal data for apple orchard monitoring. Real-time data acquisition is done from ground sensors as well as drone fitted cameras. The ground sensors monitor soil moisture, pH levels, nutrient composition (nitrogen, phosphorous, and potassium), ambient temperature, and humidity, while ESP32-CAMs fitted on drones capture images of apples, leaves, and trees. IOT enabled Unmanned Aerial Vehicle (UAV) as well as the ground sensor framework feed the data to the cloud. YOLOv8 and ResNet152 have been used to process the images for classifying the health of the apple plants. Machine learning models predict the farm yield using the ground sensor data reflecting the soil conditions. Our framework fares better than prior art in terms of accuracy. Although existing literature exhibits processing of soil data for prediction of health and yield, our study takes into consideration three further nutrient components - nitrogen, phosphorous, and potassium. Our study shows accuracy of 98.19% (apple counting) 53.66% (apple classification) 96.00% (leaf classification). To the entirety of our understanding, this study is the first to use multi-modal data inclusive of extended soil nutrients.