Learning Complementary Instance Representation with Parallel Adaptive Graph-Based Network for Action Detection

Temporal action detection (TAD) aims to find action boundaries in untrimmed videos. Video sequences contain multiple actions with various durations, which brings about great challenges to accurate boundary location. Many methods are usually ineffective in multi-scale issues in complex scenes. Besides, there is a crucial fact that local information is vital for clear boundaries. However, the traditional convolution receptive field is limited in local methods and lacks critical frame-level attention. To deal with these problems, we propose the Parallel Adaptive Graph-Based Network (PAGN), which constructs a multi-branch parallel subnetwork that retains multiple video resolutions and enables information interaction between different levels. This results in feature outputs that can represent precise location information and rich semantic information simultaneously, making it more efficient and adaptable to changes in action scale. Additionally, we also propose a novel Complementary Graph Module (CGM) that assigns differential attention to different numbers of neighbors at the current timestamp. Extensive experimental validations are conducted on the challenging datasets ActivityNet-1.3 and THUMOS-14, respectively, and PAGN all consistently exhibit significantly better performance.