In the digital age, video plays an increasingly important role because it has richer features than images. Video classification algorithm is the basis of many video understanding algorithms. In this paper, a video classification method combining image graph convolutional network (GCN) and Contrastive Language-Image Pre-training method (CLIP) is proposed to improve the feature expression of video content by integrating the visual semantic features between frames. The similarity between visual features and text classification features is measured to determine the category of video. Firstly, this paper analyzes the current status of video classification technology, and discusses the advantages of graph convolutional network and CLIP algorithm in processing multi-modal information. Then, a framework based on GCN and CLIP algorithm is designed to complete the video classification task. By constructing a relationship graph between video frames, the method can effectively capture dynamic changes in time series and combine this information with static visual features to enhance classification performance. Experimental results show that compared with traditional video classification methods, the proposed method has higher classification accuracy and better generalization ability on multiple standard data sets.