Transfer learning is the subcategory of machine learning, in which pre-trained models are used as initial models for training on a new dataset. Within-domain Transfer Learning (WTL), a subset of this approach, aims to transfer the learning to another dataset within the same domain. This paper aims to evaluate the impact of WTL in the context of Arabic handwritten character recognition. In the first step, we train four lightweight models from scratch—MobileNet, ShuffleNet, MnasNet, and SqueezeNet—on the IFHCDB database where the models perform very well. The pre-trained models are then retrained on the AHCD and Hijja datasets using the full fine-tuning technique. Compared to training from scratch, WTL improved the average training accuracy by 31.7% on AHCD and 24.6% on Hijja. In WTL, models converged faster and models with non-convergence issues were allowed to converge. Moreover, WTL reduced the average overfitting ratio by 10.9% on AHCD and 0.7% on Hijja. The obtained results highlight the promise of WTL to improve Arabic handwriting recognition and similar problems.