Osteoarthritis (OA) of the knee is the most prevalent joint disease that may cause disability. The Kellgren-Lawrence rating system is widely used to classify the severity of OA and is often used to diagnose it by interpreting X-ray imaging. The goal of this research was to create a model that uses deep learning that could automatically identify the degree of severity of knee OA, since diagnosing the condition required expertise and time. First, we examined the performance of 10 cutting-edge deep learning models. Individual state-of-the-art models were able to obtain the maximum accuracy of 0.69. Second, to solve the class imbalance, we adjusted the training technique, which increased the accuracy to 0.70. Lastly, shallow neural networks and majority voting were used in ensemble models. Our ensemble model Knee X-Net produced the best accuracy, delivering a precision of 0.72 that compares well with the most advanced techniques.