Transfer learning plays a critical role in addressing the challenges of limited training data and restricted computing resources. Pre-training enables models to learn general feature representations, and fine-tuning adapts the pre-trained weights to downstream tasks. However, designing suitable fine-tuning strategies often requires extensive manual trial-and-error or exhaustive grid search, which is time-consuming and prone to overfitting when data are scarce. To address this challenge, we propose an evolutionary computation-based framework to optimize task-specific fine-tuning strategies. The framework encodes fine-tuning strategies as chromosomes and employs a tailored genetic algorithm with novel crossover and mutation operators, alongside a performance predictor to guide the evolution towards promising regions with minimal extra training cost. Experimental results on four benchmark datasets demonstrate that the proposed method outperforms hand-crafted strategies and peer adaptive fine-tuning methods in terms of classification accuracy. Further analysis reveals the effectiveness of the newly designed operators and performance predictor, as well as the necessity of task-specific fine-tuning strategies. This study bridges the gap between evolutionary computation and transfer learning by introducing an automated framework for optimizing fine-tuning strategies, offering a practical tool for improving task-specific adaptation.