Due to the limitation of real-world scans and object occlusion, point cloud completion which aims to predict complete representation of an object from partial input has received great attention. Most existing methods rely on autoencoding architectures to recover comprehensive shape, which unfortunately limits their capability in detail expression. Moreover, they overlook outliers in partial components that often destroy the local geometries of their neighboring structural points, thereby making it difficult to rationally estimate the similarity between the alike structures locating at different regions in latent feature space. To address these issues, we propose to associate part segmentation labels with point cloud completion. Importantly, we propose an MSA (Masked Set Abstraction) module that employs the part segmentation labels as component masks and combines them with CAS (Component Attention Strategy) to exploit the intrinsic relationships among different components. With the partial property guidance from internal relevance, our framework is able to coarsely infer the complete point cloud. In addition to the feature extraction of each component, we introduce a global offset estimation method under the same guidance to predict the point-wise displacement for refining the coarse point cloud. We conduct extensive experiments to demonstrate the superiority and robustness of our method, which takes the underlying relations between different structural components into account. In comparison with state-of-the-art completion approaches, our model presents competitive performance on two benchmark datasets. The L1 Chamfer Distance scores of our model on two benchmarks have improved by at least \(2.5\%\) and \(5.6\%\) respectively.