Back to the Future: Models as Active Learning Surrogates for Next Generation ML Deployments

Rapid development of hardware goes hand-in-hand with the advancement of modern computer vision (CV) algorithms. In a typical machine learning operations (MLOps) flow, this continuous evolution of hardware and software is coupled with an active growth in data collected for training. These three pillars of MLOps continue their parallel continuous integration and improvement after an iteration of the deployment has been released. Ideally, the data chosen to improve the next iteration of the deployment is tailored for the future software solution and the future hardware capabilities which enable it. However, here we have a causality problem, where data needs to be collected for a future algorithm from a fleet of deployments which are still running the last iteration of software and hardware. In this paper, we prove that models of previous MLOps iterations are capable surrogates for choosing data for future network architectures running on more capable hardware. We show that surrogate models for the DeepLabv3+ architecture using a ResNet-50 backbone provide a +3.2 p.p. mIoU improvement on average using uncertainty scores over randomly selecting data to train the deployment model on the CityScapes dataset. Further, we show that the type of surrogate has a huge impact on the prediction capability of the deployment model. For instance, the prediction capability of a deployment model, DeepLabv3+, using a MobileNetV3 backbone, can vary by up to +2.4 p.p. on the CityScapes dataset.