The livestock industry plays a crucial role in sustaining agricultural production and rural economies. Monitoring cattle health, however, presents significant challenges on large farms where traditional methods require diagnosing each animal individually. Lameness is a major issue affecting cattle health, leading to decreased production performance on many farms. Timely detection of lameness is essential for providing effective early treatment. In this study, we propose a system using specialized depth cameras to monitor and analyze cattle back information for classifying lameness scores. We employ Detectron2 for cattle detection and segmentation, and the Intersection over Union (IOU) method for tracking, focusing solely on the cattle’s depth region. We extract various features from the cattle’s back depth data and utilize three different machine learning algorithms: K-Nearest Neighbor (KNN), Gradient Boosting, and Extra Trees for lameness score classification. The models KNN, Gradient Boosting, and Extra Trees showed strong training and validation results. Testing showed Extra Trees performing well with 88.2% morning and 89.0% evening accuracy. Our approach demonstrates the potential of depth camera in effectively classifying lameness scores, offering significant implications for livestock health management. This method not only improves the efficiency and accuracy of health monitoring in large-scale farming but also provides a practical solution for early detection and treatment of lameness, thereby enhancing overall farm productivity.