Phase retrieval (PR) aims to recover original phase signals from intensity-only measurements, which is a typical inverse problem in computational imaging. In recent years, deep learning algorithms have demonstrated considerable potential in dealing with such issues. However, the convolutional neural networks (CNNs), which are the most widely used modal, have suffered from certain inherent limitations. For example, the fixed receptive field restricts the ability of conventional CNNs to capture global dependencies, while treating the network model as a black box hinders the interpretability of the modal. Moreover, these methods only work with precisely labeled images built from expensive and ultra-precise sensors. Unfortunately, slight system aberrations can easily break the dependency between the images and the labels, making the training process extremely challenging. To address these issues, we propose PhaseNN, an unsupervised physics-driven wavefront phase retrieval network. PhaseNN adopts an encoder-decoder architecture, where the encoder efficiently extracts image features by fusing spatial and frequency domain features through the designed spatial-frequency block based on the property of the Fourier transform, which captures the global dependency and perceives the receptive field size of the image. The decoder generates pseudo-labels based on the physical optical imaging model, incorporating physical constraints into the training process. This study is the first attempt to explore the combination of spatial and frequency information in PR tasks. Experimental results demonstrate that PhaseNN outperforms existing CNN-based methods and achieves comparable wavefront phase reconstruction performance to supervised learning methods under the aberration-free condition. PhaseNN also exhibits superior robustness against the static and dynamic aberrations inherent to optical systems, thus showcasing exceptional performance.