Text Chunking to Improve Website Classification

Website classification is a crucial task in various applications such as web search, content filtering, and recommendation systems. Effectively categorizing long web pages into different categories based on their content is essential for providing accurate and personalized user experiences. Traditional transformer-based models, such as BERT and RoBERTa, have significantly advanced the field of natural language processing. However, such models face limitations when handling long sequences due to their fixed-length input restrictions resulting from their quadratic complexity. This paper presents a simple weighted stratified split approach (WSSA), to address the limitations of BERT and RoBERTa, in processing long text sequences for website classification. WSSA consists into chunking web pages into smaller chunks, then a new train chunk dataset is generated by a weighted stratified split following the distribution of the categories in the whole chunk dataset. This train chunk dataset is then used to train the models. Our approach improves the accuracy of BERT and RoBERTa models, surpassing the performance of Longformer and BigBird models. The proposed solution enables efficient processing and data augmentation, with reasonable fine-tuning times for BERT and RoBERTa models. Inference times remain efficient, showcasing the practicality of these models in real-time website classification tasks. The combination of WSSA with the index web page performs exceptionally well, highlighting its effectiveness in addressing the long text sequence limitation and improving transformer-based models for website classification.