Convolutional neural network (CNN)-based classification usually proceeds by fine-tuning a deep model pre-trained with universal large-scale source data using small-scale target data. When the size of the target dataset is limited, the classification performance depends primarily on the selected CNN. To this end, contrastive learning (CL) may be utilized for domains that lack labeled data, but have sufficient unlabeled data. In this study, we explore an approach to optimize the performance by selecting the best model for the domain to be processed as an encoder when designing a CL. Our framework employs ImageNet-pretrained lightweight CNNs as candidates for the encoder. After measuring the transferability/separability scores of the embedded features extracted through the pretrained encoder, we choose one model that best fits the target task based on the achieved values. To verify the selection strategy, we adopt a statistical comparison technique, such as Friedman test, instead of taking an average. The experiment was performed with seismic patch data, as well as two-class subsets configured on benchmark data, such as CIFAR-10/100, MNIST, FashionMNIST, and DTD. The results indicate that the classification can be optimized when using the selected CNN model as a fixed feature-extractor by referring to the score of the separability rather than the transferability. Applying the approach to various real-world scenarios, finding the best separability metric for each case, and theoretical insights into the selection remains open.