Many unsupervised salient object detection methods rely heavily on handcraft visual priors. Existing deep learning-based models require task-specific training and high annotation costs, limiting their generalization to complex scenes. In this paper, we propose a text-driven salient object detection framework by innovatively integrating the rich visual-language semantic information of Contrastive Language-Image Pre-training (CLIP) and the segmentation power of the Segment Anything Model (SAM), which is without explicit task-specific training and manually labeling. Specifically, to mitigate the negative impact of some ‘global’ patches in the final visual feature from the CLIP visual encoder, we propose a Multi-level Self Cosine-Similarity Correction model (MSCC), which calculates the cosine similarities of multi-level visual features for enhancing the local semantic correlation in saliency regions. With the modified final visual feature, we derive coarse salient regions. Then, we introduce a Multi-level Saliency Mask Refinement model, where coarse saliency maps from CLIP generate diverse prompt constraints (points, boxes, masks) for SAM, resulting in multi-level fine-grained saliency masks without manual intervention. Experimental results on public salient object benchmarks demonstrate the effectiveness of the proposed text-driven framework in segmenting salient objects, which provides empirical insights and key breakthroughs for leveraging foundation models in perceptual tasks through text prompt-based methods.