Predicting the future trajectories of agents in complex traffic scene is one of the key issues in autonomous driving, requiring reliable and effective predictions for all agents in the scene. Existing trajectory prediction models have achieved high performance on public datasets, but deploying models on vehicles requires both high accuracy and fast computation. It is necessary to balance the complexity of computation and the effectiveness of the structure when designing model. To address the above problem, we proposes a lightweight trajectory prediction model HHATP. Our method is scene-centric and located in the same coordinate system. We use different encoders for the heterogeneous scene objects and the encoded results are then fed into a hierarchical attention module, which considers both global and local interaction to model the relationships between elements. Subsequently, a dynamic weight decoder is used to obtain the trajectories of all agents. Our method achieves good accuracy on the Argoverse dataset and enables fast inference.