This paper presents a novel approach to butterfly tracking using Kalman Filters that integrates motion with its orientation and deformation dynamics to improve accuracy. By incorporating the butterfly’s deformation model and orientation, we improve the prediction of its position and its oriented bounding box dimensions, a challenge not addressed by traditional tracking methods based on Kalman filters. The performance of different Kalman filters is evaluated on spatio-spectral image sequences, showing that the Extended Kalman Filter, which accounts for non-linear deformation, gives the best results in terms of accuracy. A cost function for associating Kalman filter trajectories with new target detections is also introduced, considering position, orientation, and oriented bounding box dimensions to enhance identity preservation. This approach improves tracking performance and lays the foundation for more accurate butterfly pest recognition using raw spatio-spectral images in agricultural fields.