<p>With advancements in digital imaging, the demand for large field-of-view (FoV) imaging has been steadily increasing in various domains, including autonomous driving, augmented reality, and video surveillance. The hybrid refractive-diffractive lens, which combines the high focusing efficiency of refractive lenses with the design flexibility of diffractive lenses, holds significant promise for developing large FoV imaging systems. However, the popular deep optics faces two major challenges in globally optimizing the hybrid refractive-diffractive lens and the image restoration algorithm in an end-to-end manner. First, existing direction-independent degradation models fail to accurately compute the additional phase introduced by refractive lenses, making them inadequate for describing hybrid refractive-diffractive lenses in large FoV imaging. Second, Transformer-based restoration algorithms do not consider the degradation property of lenses, thereby limiting the quality of restored images. In this paper, we propose a differentiable end-to-end optimization framework to address these challenges, including a direction-dependent degradation model for hybrid refractive-diffractive lenses and an FoV-guided image restoration algorithm that accounts for quality variations among measured patches. Specifically, we integrate the angular spectrum method (ASM) with ray tracing to compute the additional phase of the refractive lens, facilitating the computation of the point spread function (PSF) for different FoVs. Furthermore, we encode the quality variation as a unidirectional guidance relationship within the Transformer to mitigate errors in restoration guidance. Based on the optimization results, we develop a physical system and validate the effectiveness of the proposed method in both simulation and real-world scenes.</p>