We study the flavor structures of zero-modes, which are originated from the modular symmetry on \( {T}_1^2\times {T}_2^2 \) and its orbifold with magnetic fluxes. We introduce the constraint on the moduli parameters by τ2 = Nτ1, where τi denotes the complex structure moduli on \( {T}_i^2 \) . Such a constraint can be derived from the moduli stabilization. The modular symmetry of \( {T}_1^2\times {T}_2^2 \) is \( \textrm{SL}{\left(2,\mathbb{Z}\right)}_{\tau_1}\times \textrm{SL}{\left(2,\mathbb{Z}\right)}_{\tau_2}\subset \textrm{Sp}\left(4,\mathbb{Z}\right) \) and it is broken to Γ0(N) × Γ0(N) by the moduli constraint. The wave functions represent their covering groups. We obtain various flavor groups in these models.