In recent years, Convolutional Neural Networks (CNNs) and Graph Convolutional Networks (GCNs) have been widely used for hyperspectral image (HSI) classification, with CNNs specializing in capturing local receptive features and GCNs focusing on structural features. To leverage the strengths of both CNNs and GCNs for achieving more robust classification performance, several fusion networks that combine CNNs and graphs have been proposed for HSI classification. However, existing CNN-based methods struggle to capture high-order interactions among different spectral bands. However, GCN-based approaches often rely on fixed or simplistic graph models for feature learning. Therefore, in this paper, we proposed a fusion network composed of a superpixel-level branch based on adaptive hypergraph networks and a pixel-level convolution branch based on pyramid spatial attention and partial channel convolution. The high-level features of the HSIs obtained from these two branches are adaptively fused. Experimental results in four datasets demonstrate that our proposed network model exhibits significant advantages compared to existing convolutional neural networks, graph convolutional networks, and dual-branch fusion networks.