Wildlife intrusions on farmlands result in substantial crop losses and financial hardship for farmers, particularly for those residing within close proximity of forests and wildlife areas. In this paper, a new multimodal deep learning (DL) architecture has been formulated with a proper combination of RGB, thermal and acoustic data, but also with the additional use of accelerometer/movement sensors as part of the pipeline to identify intrusions in a timely, accurate manner. The model architecture is made up of a combination of CNNs to extract features, an attention-based fusion model, temporal analysis (Temporal Convolutional Networks, TCN Layer) to include the time context of the intrusion, and is implemented on edge devices, giving it superior real-time capabilities. The proposed model can achieve 94.5% Accuracy, 95.2% precision, 93.7% recall, and F1 Score of 94.4% with a false alarm rate of 3.1%. These preliminary results indicate the considerable advantages over the three baseline models, which included YOLOv5 and PIR-SVM systems, and promising results compared to spatial, temporal, and spectral detection methods. The proposed scalable solution would be robust enough, not only to issue automated alerts, but also allow for active learning, and represent a viable option for widespread deployment at farm-level, where smart farming in areas vulnerable to wildlife based intrusions still has great potential.