Prompt Tuning (PT) efficiently adapts pre-trained models to downstream tasks. Recent studies have suggested incorporating prompt tuning into the framework of federated learning. This paper aims to answer “whether it is necessary to seek federation when clients already possess strong few-shot learning abilities with local prompt tuning” through experimental studies. We simulated various types of data distribution shifts that may exist among clients in real-world applications and compared the performance of federated PT with its local counterpart. Our results show that the “gain window” of federation in prompt tuning is generally smaller than full-model federated learning, and is directly impacted by the cross-client distribution shift and the amount of data on each client. We have found that implementing federation directly to PT in cross-domain situations can lead to performance degradation, calling for more careful considerations and advanced formulations to ensure optimal results.