Multi-view graph clustering (MVGC) has made great progress in analyzing the interaction patterns of complex networks. Existing methods leverage different graph filters to obtain high- and low-pass signals and implement multi-view fusion. However, these filter-based methods face a scalability issue, which results in insufficient representation discrimination. Besides, they lack view-specific semantics in multi-view fusion, leading to poor information fidelity. To address these limitations, we propose a graph clustering framework with scalable graph filters and view-specific semantic fusion (SGSF-GC). SGSF-GC designs a Beta-based scalable graph filter and cohesion-based fusion mechanism to capture and integrate high- and low-frequency signals. Then SGSF-GC employs class activation mapping to capture semantics of view-specific representations for multi-view fusion. Finally, SGSF-GC conducts KL-based graph clustering. Extensive experiments on five public datasets with eleven baselines verify the utility and superiority of SGSF-GC.