Open-vocabulary semantic segmentation models often fall short in real-world applications due to accuracy limitations. Conversely, category-agnostic models achieve higher accuracy thanks to their large-scale and low-cost training data. Thus, we aim to combine them together for accurate open-vocabulary semantic segmentation. To achieve this, we propose an Open-Vocabulary Self-Interactive semantic Segmentation model (OVSI-Seg). It uses an open-vocabulary semantic segmentation model (as a leader) to guide a category-agnostic image segmentation model (as an executor) to segment the objects. This idea is inspired by the collaboration paradigm between humans, where human leaders provide promising directions and human executors finish high-quality works. We design a leader-to-executor interaction to guide the executor model to improve its unsatisfactory mask through contrastively sampling point prompts from the leader’s mask. We also design an executor-to-leader interaction to correct the leader model’s mistakes by improving its category embeddings with the executor’s mask. Akin to human discussions, we allow the leader and executor to have multi-round interactions and track improvements to determine the optimal stopping time. The plug-and-play nature of OVSI-Seg preserves the generalizability of pre-trained models, making it require no additional training and compatible with other techniques. Despite its simplicity, OVSI-Seg outperforms the state-of-the-art models on the PASCAL-5i, COCO-20i, and DUTS benchmarks. The code is available at https://github.com/XinshuangL/OVSI-Seg .