Transfer learning is widely used lately to train on small datasets using pre-trained models. The process aims to reuse the parameters of a trained model on a source task and fine-tune only specific layers on the target task. However, the efficiency of transfer learning depends on the appropriate selection of knowledge from the pre-trained model, relevant to the specific task, which remains a significant challenge for this approach. The most commonly used approach for transfer learning is fine-tuning, which retrains certain parameters of the pre-trained model on the target task dataset. At present, the selection criteria are still a manual process. Therefore, in this study, we propose an approach to adjust pre-trained models for a specific task by selecting automatically the most effective layers in the source models to be re-trained, to optimize transfer learning. The performance was evaluated with 8 public classification datasets. Our proposed approach outperformed standard fine-tuning baselines, and the results demonstrated that we achieve the performance of layer selection methods baseline while requiring less computation.