Nanocrystalline system of pristine Ho2O3 (HO) and Co-doped Ho2O3 (Ho1.9Co0.1O3-δ, HCO) were synthesized by chemical co-precipitation method. These samples were investigated by analyzing structural, morphological and magnetic data. To enhance the oxygen vacancy, both HO and HCO samples are annealed in argon atmosphere at 700 \(^\circ{\rm C}\) for 6 h. Rietveld analysis of X-ray diffraction (XRD) pattern of both samples confirmed the desired crystallographic phase and successful substitution of Co-ion in Ho2O3. Microstructural analyses of prepared samples are also performed by Rietveld refinement method. Energy dispersive X-ray (EDAX) and X-ray photoelectron spectroscopy (XPS) analysis confirm that impurity phases like Co3O4 and CoO are absent in HCO. Also, XPS analysis indicates that mismatch of ionic valence state of dopant ion Co2+ and host cation Ho3+ introduces oxygen vacancy in HCO. Magnetization (M) as a function of applied magnetic field (H) of both bare and doped samples are recorded at different temperatures as a function of magnetic field in the range of ~ − 5 T to 5 T. Nonlinearity in M-H loops recorded at ~ 50 K indicates the presence of magnetic ordering below ~ 50 K for both samples, but the signature of ordering in doped sample more prominent than that of the pristine. Good fitting of magnetization vs. temperature curve below ~ 50 K by the combined equation of 3D spin wave model and Curie–Weiss law and lack of saturation of M-H loop below ~ 50 K indicates coexistence of paramagnetic (PM) and ferromagnetic (FM) phases in both samples. The observed magnetic phase transition is explained by vacancy mediated bound magnetic polaron (BMP) model. Also, antiferromagnetic transition in the doped sample is observed at 3 K.