Detecting hate speech online has become increasingly important due to the surge in harmful content on social media. This is particularly challenging for resource-constrained languages like Bengali. This paper presents a dataset specifically created for detecting contextual hate speech in Bengali, developed through extensive data collection, preprocessing, and both manual and automatic labeling. It comprises 15,000 annotated texts categorized into hate speech and non-hate speech, with a Cohen’s kappa score of 0.88, reflecting strong agreement among annotators. We assessed the dataset using machine learning (ML), deep learning (DL), and BERT-based models. Among these, the BERT-based model XLM-R excelled, attaining an F1 score of 0.94 and an accuracy of 0.92 when context was considered, and an F1 score of 0.89 with an accuracy of 0.87 without context. These findings highlight that integrating context notably enhances the accuracy of hate speech detection, contributing to more effective methods for identifying and mitigating harmful online content.