Social media connect individuals across vast geographical distances, fostering free expression. However, it can be undermined by online harassment, often manifested in comment sections, causing emotional distress, reputational and even physical harm. Automated systems are developed to detect and remove these harmful contents but overlook the concerns of bias and losses in the pursuit of higher accuracy. This study addresses cyberbullying detection in Bangla text, a widely spoken language. We leverage BanglaBERT (small) for the classification task using a large publicly available dataset consisting of 44,001 comments. After analyzing the data, we applied rigorous data-cleaning procedures along with pre-processing techniques that removed non-linguistic artefacts. Our proposed model demonstrated a remarkable binary and a multi-level classification accuracy of 94.91% and 90%, respectively, while minimizing overfitting complications and maintaining minimal loss. This performance opens a new avenue for future research in low-resource scenarios for real-world applications.