Industrial Anomaly Detection (IAD) aims to identify and locate anomalies in images, which is crucial for industrial manufacturing. Traditional unsupervised methods, which rely exclusively on normal data, produce simple classification results and rough anomaly localizations with manually defined thresholds. Meanwhile, supervised methods often overfit to prevalent anomaly types due to the scarcity and imbalance of anomaly samples. Both paradigms suffer from the “One Anomaly Class, One Model” issue, which complicates practical applications. To tackle these issues, we propose Referring Industrial Anomaly Segmentation (RIAS), a novel paradigm that leverages language to guide anomaly detection. RIAS offers two key benefits: it generates precise, fine-grained masks directly from textual descriptions without the need for manual threshold adjustments, and it employs universal prompts to detect various anomaly types with a single, unified model. To support RIAS, we introduce the MVTec-Ref dataset, designed to mimic real-world industrial environments with diverse, scenario-specific referring expressions. This dataset is distinguished by two primary features. Firstly, the anomalies are unevenly distributed in terms of size, with small anomalies comprising 95% of the dataset. Secondly, in contrast to natural images that typically prioritize object recognition, the anomaly images in this dataset are designed to focus primarily on detecting the patterns of anomalies. To evaluate the effectiveness of our paradigm and dataset, we propose a benchmark framework called Dual Query Token with Mask Group Transformer (DQFormer), which is enhanced by Language-Gated Multi-Level Aggregation (LMA). The LMA module enhances visual features at multiple scales, improving segmentation performance for anomalies of varying sizes. Additionally, unlike traditional query-based methods that rely on redundant queries, which are not well-suited for anomaly-focused images, our novel query interaction mechanism in DQFormer uses just two tokens, i.e. Anomaly and Background. This design facilitates efficient integration of visual and textual features. Extensive experiments demonstrate the effectiveness of our RIAS in IAD. We believe that RIAS, equipped with the MVTec-Ref dataset, will push IAD forward open-set.