Let π: (X, T) → (Y, S) be a factor map between two topological dynamical systems, and \(\cal{F}\) a Furstenberg family of ℤ. We introduce the notion of relative broken \(\cal{F}\) -sensitivity. Let \(\cal{F}_{s}\) (resp. \(\cal{F}_{\text{pubd}},\cal{F}_{\text{inf}}\) ) be the families consisting of all syndetic subsets (resp. positive upper Banach density subsets, infinite subsets). We show that for a factor map π: (X, T) → (Y, S) between transitive systems, π is relatively broken \(\cal{F}\) -sensitive for \(\cal{F}=\cal{F}_{s}\) or \(\cal{F}_{\text{pubd}}\) if and only if there exists a relative sensitive pair which is an \(\cal{F}\) -recurrent point of (Rπ, T(2)); is relatively broken \(\cal{F}_{\text{inf}}\) -sensitive if and only if there exists a relative sensitive pair which is not asymptotic. For a factor map π: (X, T) → (Y, S) between minimal systems, we get the structure of relative broken \(\cal{F}\) -sensitivity by the factor map to its maximal equicontinuous factor.