Accurately predicting the future motion of agents is crucial for human-robot interactive systems such as autonomous driving. However, in real-world traffic scenarios, complex social interactions make accurate trajectory prediction challenging, especially considering the different interaction patterns between different types of agents. Existing trajectory prediction methods usually ignore the heterogeneity of interactions, and treat all interactions equally. In this work, we propose HiTraj, a multimodal trajectory prediction network. A novel and powerful heterogeneous interaction learning Transformer (Hi-Transformer) is introduced, which combines prior and hidden interaction information to achieve more detailed heterogeneous interaction learning. In addition, we have introduced a simple but effective variant of multi-head attention mechanism, multi-style attention, through a series of style channels, it generates multimodal trajectories with different styles. We evaluated HiTraj on three widely used trajectory prediction datasets and the results show that HiTraj achieves comparable or superior performance compared to baseline models, especially achieving state-of-the-art performance on the SDD dataset with a large amount of heterogeneous interactions.