Lcp-mixer: a lightweight model based on concept-level perception for NLP

Transformer-based models excel in natural language processing tasks but demand extensive computational resources and memory. To address this challenge, the pNLP-Mixer, a projection-based MLP-Mixer model for Nature Language Processing, was proposed. It utilizes a parameter-free projection layer to generate text embeddings, serving as input for the MLP-Mixer, which is an all-MLP architecture. However, these models have limitations in representing the complexity of human knowledge. Human tend to use diverse and structured descriptions (e.g., Concept-Level structure) rather than a single sequential description when describing the objective world. Based on the concept-level structure, this paper proposes a lightweight model based on concept-level perception for NLP, named LCP-Mixer. It replaces the MLP-Mixer component in the pNLP-Mixer with the LCP module, abstracting the concept-level information into a Group-Block pattern. This design enhances the flexibility of LCP-Mixer, enabling it to perceive information at various concept levels and global semantic information. Evaluations across various downstream tasks illustrate the performance of LCP-Mixer. On MTOP dataset, LCP-Mixer achieves 98.07 \(\%\) $\%$ of mBERT’s performance with only 0.2M parameters, an 850 times reduction in parameters. Compared to models like pNLP-Mixer and pQRNN (projection Quasi-RNN), LCP-Mixer strikes a superior balance between parameters and accuracy. For text classification task, it achieves over 90 \(\%\) $\%$ of XLNet’s performance and outperforms pNLP-Mixer on multiple datasets. These results highlight LCP-Mixer’s ability to achieve the trade-off between computational resources and accuracy.