Bricks have long been a fundamental material in construction, appreciated for their durability, strength, and versatility. Even with advancements in modern building technologies, bricks remain a top choice due to their excellent thermal insulation, fire resistance, and diverse design possibilities. In countries like Bangladesh, bricks are classified into three grades (Grade One, Grade Two, and Grade Three), each suited for different purposes. However, traditional manual grading methods are often inconsistent and labor-intensive, necessitating more efficient solutions. This study introduces an enhanced brick grading system using a customized DenseNet-169 model, named BrickNet. By leveraging transfer learning combined with Explainable Artificial Intelligence (XAI) techniques, our method improves both the accuracy and interpretability of brick classification. We utilize Local Interpretable Model-Agnostic Explanations (LIME) and Gradient-weighted Class Activation Mapping (Grad-CAM) to provide transparent visual explanations for the model’s predictions, thereby increasing transparency. Our experiments, which include a comparison with six pre-trained deep learning (DL) models, namely DenseNet-169, MobileNet-v2, ResNet-50, VGG-19, Xception, Inception-v3, reveal that hyperpararmeter-optimized DenseNet-169 model (BrickNet) achieves a maximum accuracy of 97.33% with minimal loss. This research underscores the effectiveness of automated grading in surpassing traditional methods and offers a promising advancement for the brick classification process in the construction sector.