Early and accurate detection of knee osteoarthritis (KOA) is crucial for preventing joint degeneration and ensuring early treatment. Traditional diagnostic methods based on radiographic interpretation, such as the Kellgren-Lawrence classification system, are highly subjective and prone to inter-observer variability. In this paper, we present a hybrid deep learning approach for automatically segmenting and classifying KOA severity from X-ray images. Unlike traditional models, our system incorporates domain-specific knowledge to improve interpretability and generalizability across clinical datasets, enabling more reliable diagnoses even with previously unseen data.