Neural Radiance Fields (NeRF) has shown impressive view synthesis results by fitting Multi-Layer Perceptrons (MLPs) to 2D images captured from the 3D scene. The main drawback is that it requires training from scratch for every new scene and relies on densely captured images to train the deep network. To improve its generalization ability, recent works equip NeRF with image encoders and Multi-View Stereo (MVS) techniques, which can mine semantic and geometric features of the scene from sparse input views to help synthesize novel views. However, these extracted scene priors are not fully utilized in existing methods. To this end, this paper focuses on how to better leverage them to improve the rendering quality of generalizable NeRF. First, based on the understanding of the scene by MVS, we propose an efficient Probability-Guided Sampling (PGS) strategy to help allocate more samples around regions containing visible content. Furthermore, a self-supervised Point Visibility Estimation (PVE) module is designed to assist in better aggregating features by leveraging the visibility of samples in different views. Extensive experiments demonstrate that compared with existing generalizable NeRF models, our method achieves superior rendering quality when generalized to unseen scenes.