Let G be a finite group and p be a prime. We define \(N^{\mathcal {N}_p*}(G)\) to be the intersection of the normalizers of the p-nilpotent residuals of all two-generator subgroups of G whose p-nilpotent residuals are nilpotent. We show that \(N^{\mathcal {N}_p}(G)=N^{\mathcal {N}_p*}(G)\) . Using the method in the present paper, we will be able to give an affirmative answer to an open problem in Shen et al. (Mediterr J Math 19:191, 2022), which also indicates that similar conclusions hold for many formations. It is also proved that \(G=N^{\mathcal {N}_p}(G)\) if and only if every three-generator subgroup H of G satisfies \(H=N^{\mathcal {N}_p}(H)\) . To this end, we introduce and investigate the IO- \(N^{\mathcal {N}_p}\) -groups, i.e., the groups G such that \(G\ne N^{\mathcal {N}_p}(G),\) but each proper subgroup and each proper quotient of G equals its p-nilpotent norm. Moreover, new results in terms of the p-nilpotent norm and the p-nilpotent hypernorm \(N^{\mathcal {N}_p}_\infty (G)\) are given.
