Deformable image registration is crucial for medical imaging tasks such as disease diagnosis and treatment planning. Recent advances have relied on either transformer- or attention-based deep learning registration architectures for dense deformation field estimation, resulting in high complexity but with insufficient granularity. Moreover, the insufficient capability of multi-scale resolution modeling still poses a challenge to the accurate and effective registration of large volume deformations. Therefore, we propose a Recursive Wavelet-driven Network (RWNet) by integrating multi-level wavelet sub-bands and a recursive strategy to facilitate multi-scale deformation representation learning for deformation field prediction. Specifically, a pure convolutional encoder with discrete wavelet transform is developed for deep feature mining at multiple scales with different frequency components without high-weight attentions. Then, a deformation fusion-based field estimation method is carefully designed, which combines the frequency-driven field with the spatially-enhanced one to facilitate the reconstruction of the displacement field. Finally, a step-by-step recursive strategy is adopted by integrating high-level features to iteratively refine transformations in a coarse-to-fine manner. Extensive experiments on two publicly available brain MRI datasets demonstrate the superior performance against existing registration methods.