This paper deploys deep learning models to detect and categorize knee osteoarthritis disease. The mentioned disease affects the joint areas and is marked by degrading transformations in the surrounding bone structure and tissues accompanied by a progressive breakdown of articular cartilage. In such studies, the Kellgren–Lawrence (KL) grading system is mainly adopted to determine the extent of knee osteoarthritis (OA). Based on the training of previously graded X-ray images, deep learning models can simultaneously automate the prediction process and expedite the diagnosis process with reduced human error. Our proposed model integrates a modified version of the Double U-Net model, a hierarchical system, and CBAM, an attention module to accurately predict the KL class of the knee OA X-ray scans. The training of the proposed model stands at 98%, while the validation accuracy is 80%. Our proposed model is efficient and achieves the desirable outcome with the available datasets.