Document Image Dewarping (DID) task aims to address the issue of geometry distortion and improve image quality. In this paper, we propose a simple but effective method, named DocHFormer, that can take hierarchical priors features of images, including document image mask and coordinate positions, as additional information to realize accurate representation. To better exploit these fused information for dewarping, we take them into a harmonized space random shuffle operation, which can stochastically rearrange the pixels across spatial space and further use inverse operation to recover the original order. This way can adapt to allocate each feature pixel with equal probability and thus make full use of multi-type features. Furthermore, we introduce this mechanism into local self-attention to use linear complexity to input resolution and also design a new feed-forward network with structural modeling to boost representation. With the help of the above components, our proposed DocHFormer can achieve competitive performance with lower complexity and also outperform the existing state-of-the-art on several popular datasets.