FedInc: One-Shot Federated Tuning for Collaborative Incident Recognition

Social media on mobile devices is an essential channel for monitoring emergency events, including floods, fires, snowstorms, hurricanes, car accidents, etc.. Serving as a low-latency data source, they provide a crucial avenue for understanding the progression and consequences of disasters. Therefore, recognizing these incidents from images on social media is a highly valuable research problem. Traditional methods involve collecting raw images from a large number of sources to a central server for model training. However, this approach requires significant communication costs and poses the risk of compromising user privacy. To address these challenges, this paper proposes a collaborative recognition method named FedInc based on one-shot federated tuning. FedInc enables participants from different regions to collaboratively establish a global recognition model without sharing raw local data. Specifically, FedInc considers a practical setting where the server contains data of some classical categories, while the clients contain numerous incident categories missing on the server. The server initially utilizes existing data to perform LoRA fine-tuning on a pre-trained Vision Transformer (ViT), obtaining a base model, and then sends the fine-tuning parameters to the clients. During client training, FedInc employs a two-stage fine-tuning approach to alleviate the client’s forgetting of server categories. Additionally, during the server aggregation phase, FedInc designs a new aggregation strategy for LoRA parameters. Extensive experiments demonstrate that FedInc significantly improves the accuracy of the server’s base model and outperforms comparative methods.