Fighting detection helps maintain order and safeguard the lives of people in public. Using video surveillance to detect fighting in public places is a common and effective solution. Conventional detection methods focus on analyzing the angles between joints, striking, kicking actions, etc., of individuals engaged in fighting. These methods predominantly capture the action features of individuals, without considering the variation patterns of the motion trajectories during fighting. Moreover, this research faces a significant challenge: many non-fighting behaviors such as hugging, running and dancing may visually present similar action features to fighting. This similarity makes it difficult to accurately differentiate between fighting and normal behavior in practical applications. In contrast, during fighting, individuals exhibit intense motions and keypoints’ motion trajectories lack obvious regularity, which provides higher discriminability compared to other behaviors. Based on this, we proposed an effective and practical detection method that analyzes the motion trajectory patterns of different individuals’ keypoints over a period of time to identify the distinct patterns between fighting and normal behaviors. The steps are as follows: firstly, YOLO-Pose is employed to estimate the human pose, obtaining bounding box and individuals’ keypoints, in which centroid of the bounding box is considered one of the keypoints; secondly, DeepSORT is utilized to track individuals, acquiring the motion trajectories of keypoints; thirdly, using variance to identify the individual with abnormal speed variations in keypoints’ motion; finally, the entropy of motion directions of abnormal individuals’ is used to detect fighting behavior. The algorithm achieved 91.1% accuracy and 91.4% specificity on the RWF-2000 dataset, 95.9% accuracy and 95.7% specificity on our custom dataset, whereas the rate of false detection is much lower.