G-invariant Bergman kernel and geometric quantization on complex manifolds with boundary

Let M be a complex manifold with smooth boundary X, which admits a compact connected Lie group G acting holomorphically and preserving X. We establish a full asymptotic expansion for the G-invariant Bergman kernel under certain assumptions. As an application, we get G-invariant version of Fefferman’s result about regularity of biholomorphic maps on strongly pseudoconvex domains of \({\mathbb {C}}^n\) ${\mathbb{C}}^n$ . Moreover, we show that the Guillemin–Sternberg map on a complex manifold with boundary is Fredholm by developing reduction to boundary technique, which establishes “quantization commutes with reduction” in this case, as an analogue of its CR version (Hsiao et al. in Commun Contemp Math 25(10):2250074, 2023, Theorem 1.2).