Wildlife movements on farmland pose a significant threat to crop production, farmer safety, and farming equipment in remote, unmanned settings. More traditional surveillance measures involving barriers and motion detector sensors are limited by, among other things, false alerts and lack of awareness of the surroundings. This study presents a deep learning (DL) framework to enhance the detection of wildlife movements by using RGB imaging, thermal imaging, and motion detection. By employing lightweight models such as YOLOv5-Nano and MobileNetV3, the new environment incorporated spatial-temporal attention and a background suppression scheme to ameliorate the detection. These architectures were optimized for edge deployment with TensorRT and quantization-aware training. The field testing of the proposed multi-modal architecture demonstrated high performance at 96.5% accuracy, 94.8% precision, and 4% false alarms and excellent real-time performance demonstrating latency of 82 ms/frame, and energy consumption of 3.6 W/h. These results highlight the promise of an accurate deep learning-based multimodal architecture for real-time wildlife detection, and set the stage for scalable smart agriculture applications.