Task Prioritization for Remote Sensing AI Models
摘要
Improvements in self- and semi-supervised machine learning (ML) afford an opportunity to train AI foundation models (FMs) for a variety of specialized sensor modalities. Because a foundation model’s ability to generalize across varied remote-sensing (RS) tasks depends on the data distribution it was trained on, different downstream applications require different levels of feature specificity in the model’s backbone. Therefore, users need a clear way to assess a model’s descriptive capacity so they can prioritize development and choose the right model for each task. Although the end goal of operationalizing an FM is likely to involve task-specific fine-tuning, as the corpus of available FMs for various sensors expands, it is not tractable to label fine-tuning sets for every possible combination of tasks and modalities; in fact, having sufficient volumes of labeled data to fine-tune every combination eliminates the benefit of self-/semi-supervision. To address this issue, we have invented a means of probing AI models’ embedding spaces themselves to gauge the specificity of their feature encodings. Our method characterizes feature spaces by comparing task performance via linear mappings of encoded features across multiple models’ embedding spaces, allowing for benchmarking of novel models against existing counterparts with fewer labeled examples. By constructing linear mappings between embedding spaces, we show that it may be possible to predict a model’s performance at a given task without labeling data in that modality or fine-tuning the model for that task. We discuss the use cases of our technique for research programs and for AI capability acquisitions.