Recent advancements have introduced hybrid models combining Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) for the classification of cervical histopathological images. This hybrid approach effectively captures both global and local contextual information, achieving commendable performance. However, the self-attention mechanism in ViTs is computationally intensive and often inadequate for cervical histopathological images due to their complexity. In this study, we propose an efficient and lightweight CNN-ViT hybrid model (CervicalFormer) to address these challenges. A novel Linear Angular Cross Covariance Self-Attention (LACCSA) module, with linear complexity in the number of tokens, is incorporated into the model. LACCSA integrates advanced Cross-Covariance Attention (XCA) and Linear-Angular Attention, enabling the model to effectively extract common features within the same class and distinguish subtle differences between classes. Additionally, a novel activation function, ExpReLU, has been designed to mitigate distribution shift in cervical histopathological image datasets. Compared to the commonly used GELU activation function in ViTs, ExpReLU is achieving superior performance. The model was tested on the LDCH dataset, where it achieved leading accuracy using the least number of parameters. Furthermore, the efficacy of ExpReLU was corroborated on another binary classification cervical histopathological dataset, CAISHI. Both LACCSA and ExpReLU can be seamlessly integrated into other ViT models. The code for CervicalFormer is publicly available at: https://github.com/507NN/CervicalFormer .