Refining Skeleton-Based Temporal Action Segmentation with Edge Information

The purpose of temporal action segmentation (TAS) is to identify the actions of each frame in the long untrimmed action sequence and classify multiple actions. Most of the existing methods analyze video data without considering the spatial information of human joints. Although the classification of single-frame action has achieved remarkable results, there are challenges in over-segmentation under high precision. To address this challenge, we propose Local Refining with Spatial-temporal Graph Convolutional Neural Network (LR-GCN). The initial prediction is generated by spatial-temporal graph convolutional. The initial predictions are then fed into two branches, one branch is used to refine the initial prediction and the other uses the Motion Edge Prediction (MEP) module to predict the edges of the action segment. Finally the output from both branches is fed together into a Local Smooth Adjustment (LSA) module, which combines the edge information with the prediction results for a more accurate classification result. Our framework is compared with state-of-the-art methods and experimental results show that our framework effectively mitigates the over-segmentation problem on TAS tasks.