Convolutional neural networks (CNNs) and Vision Transformers (ViTs) are essential in computer vision. While convolutional neural networks remain robust and are widely used in traditional computer vision tasks, ViTs are rapidly emerging as a new trend. Convolutional neural networks capture local features of images by local dependency and are computationally efficient, especially when dealing with large-scale image data. ViTs globally capture the relationships between input image sequences by self-attention. Still, the large amount of computational resources are required for the similarity computation between tokens and the computation complexity quadratically increases with the number of tokens. To address these issues, we propose a hybrid neural network architecture combining convolution and transformer, called SUMMNet, which consists of four stages, each containing two important blocks, i.e., Local Large Convolution Block (LLCB) and Global Self-Attention Block (GSAB). LLCB introduces the Large Kernel Convolution Attention (LKCA) to capture local detail features more efficiently. GSAB employs a new Lightweight Cross-Head Self-Attention (LCHSA) to enhance the interactions between heads for global abstract information, while decreasing the computation complexity with the dimensionality reduction of Key and Value in self-attention. The proposed SUMMNet has the advantages of both CNNs and ViTs in terms of efficiency and effectiveness. We evaluate our SUMMNet through extensive experiments and it shows promising performance in image classification tasks. Our SUMMNet achieves 84.1% top-1 accuracy with 11.4G FLOPs on the ImageNet-1K image classification task, surpassing Swin-Transformer by 0.6% with 36% less parameters and 30% fewer FLOPs. Our source code is available at https://github.com/YaqiLi01/SUMMNet.git .