Lung and colon cancers have low survival rates. Deep learning-based automated techniques for detecting cancer from medical images and classifying cancer to its correct type, have given a breakthrough with improved accuracy, although this generally requires a huge amount of data and time for training these models. The first goal of our research is to improve the classification accuracy of standard Transfer Learning Models by fine-tuning these models for the task of lung and colon cancer identification and classification on the LC25000 dataset (Histopathological images of lung and colon tissues). Fine-tuning the Transfer Learning models also proposes an alternate methodology to achieve better accuracy in addition to the various ensemble or hybrid techniques being used by other researchers. To achieve our first goal, we have considered 5 standard Transfer Learning models: MobileNetV2, Xception, EfficientNetB6, EfficientNetV2L, and NasNetLarge with accuracies 71.3%, 79.0%, 84.0%, 85.7%, and 82.5% respectively (as defined by Keras on ImageNet validation dataset). We trained all these models from scratch on the LC25000 dataset, fine-tuning their hyper-parameters and architecture, and re-evaluating these fine-tuned models on the measures: accuracy, f1 score, precision, and recall. Our fine-tuning of models has resulted in validation accuracies in the range of 93.96–99.68%, beating the existing benchmarks of these models as defined by Keras. The highest accuracy of 99.68% was achieved with our fine-tuned version of NasNetLarge. The second goal of our study was to propose a small-sized Convolutional Neural Network (CNN) model which is computationally faster to train as compared to the standard or fine-tuned Transfer Learning models. The proposed small-sized CNN has only 15 layers, and its performance on LC25000 was reasonably well with a validation accuracy of 94.67%. We shall be discussing the hyper-parameters and architecture of the fine-tuned transfer learning models along with our newly proposed CNN model.