Pedestrian trajectory prediction is a crucial research area in the field of autonomous driving. Most of the existing works have primarily focused on the movement patterns and interactions of individual pedestrians, without considering group behavior. To address this gap, future research should aim to understand the collective behavior of pedestrians in various scenarios. We propose a new method called XrGroup, which utilizes group behavioral patterns as physical constraints to achieve high-precision pedestrian trajectory prediction. First, a module for extracting group features is designed to group pedestrians based on their historical trajectories to obtain group features at the position level. Additionally, the module introduces the pedestrians’ speed information to effectively learn group features at the speed level. Second, we design a pedestrian trajectory prediction module based on a sparse graph convolutional network. The network extracts effective interaction features between groups and pedestrians, and uses them to infer predicted trajectories by obtaining probability distribution of future trajectories. Our model achieves excellent performance with values of 0.24/0.43 regarding the Average Displacement Error (ADE) and Final Displacement Error (FDE) results obtained on the ETH/UCY datasets, respectively.