Large Language Model for Action Anticipation

Video action anticipation is a challenging task in which the model needs to reason about the upcoming action based on historical video information. To achieve effective action comprehension and anticipation, current research primarily focuses on leveraging visual cues, such as RGB or optical flow, to extract action-related semantics and make predictions about future actions. However, the task of extracting action embeddings from visual information and predicting future action may be suboptimal, as these visual features often contain extraneous details, including environmental factors, resulting in a less discriminated action embedding. Additionally, we argue this high-level semantic reasoning task of action participation is not limited solely to the visual modality. It is possible to enhance action anticipation by incorporating language-based cues, especially considering the powerful priors in large-scale pre-trained language models. To solve this problem, we propose the Anticipative Language Model (ALM), which leverages the implicit prior knowledge in the large language model to complete the action anticipation from two perspectives: perception and knowledge. ALM achieves state-of-the-art performance on the EPIC-KITCHENS55 and EPIC-KITCHENS100 datasets. Moreover, for the first time, we verify the action anticipation task in the few-shot scenario and demonstrate our method’s superior performance.