Observed animal movement trajectories are often the result of a latent process whereby an animal transitions between discrete behavioural states such as foraging or resting. The standard approach for analysing multi-state movement data is to employ hidden Markov models, and these models have been used in a wide array of animal movement studies. Recent developments have enabled hidden Markov models to be applied to irregularly sampled data, as well as providing uncertainty quantification in the inferred latent states. However, all such models rely on the unrealistic underlying assumption that sojourn times in each behavioural state are exponentially distributed, meaning there is always a constant probability of leaving a state. Here, we propose a hidden semi-Markov model where movement is modelled as a continuous-time integrated Ornstein-Uhlenbeck process and behavioural state transitions are governed by an arbitrary distribution of sojourn times. We employ a Monte Carlo expectation-maximisation algorithm to reconstruct the hidden state sequences and obtain the posterior distribution of the latent sequences as well as efficiently optimising the parameters of the movement and state-switching dynamics. We demonstrate the effectiveness of our framework by applying it to synthetic data and to two real tracking data case studies.